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ABSTRACT 


The  proliferation  of  data  accessibility  has  exacerbated  the  risk  of  shallowness  in  information 
analysis,  making  it  increasingly  difficult  to  tell  when  analysis  is  sufficient  for  making 
decisions  or  changing  plans,  even  as  it  becomes  increasingly  easy  to  find  seemingly  relevant 
data.  In  addressing  the  risk  of  shallow  analysis,  the  concept  of  rigor  emerges  as  an  approach 
for  coping  with  this  fundamental  uncertainty — motivating  the  need  to  better  define  and 
understand  analytical  rigor.  The  concept  of  rigor  is  explored  in  this  thesis  through  a  study 
that  asks  how  professional  analysts  decide  when  there  is  sufficient  rigor  in  an  analytic 
process.  Nine  professional  intelligence  analysts  participated  in  a  scenario  walkthrough  in 
which  they  critiqued  the  analysis  processes  of  two  junior  analysts — one  representing  a  high- 
rigor  analysis  process  and  the  other  a  low-rigor  process.  In  the  study,  participants  assumed 
the  role  of  analyst  supervisor,  deciding  if  these  analyses  were  of  sufficient  rigor  to  send  to  a 
decision  maker — a  fundamental  judgment  task  characterized  as  the  Supervisor's  Dilemma. 
This  study  design  validated  and  refined  the  Elicitation  by  Critiquing  methodology,  also 
developing  the  Liquified  Natural  Gas  Scenario,  based  on  security  issues  that  challenge  safety 
analyses,  as  a  cognitive  case  for  exploring  themes  in  information  analysis.  This  research 
identified  three  general  findings  on  rigor  in  information  analysis.  First,  it  found  that  process 
insight  influenced  judgments  of  rigor.  Second,  it  found  that  while  similar  cues  were  used  in 
forming  assessments  of  rigor,  the  way  in  which  those  cues  were  interpreted  as  indicating 
rigor  tended  to  be  more  varied.  Third,  the  results  of  the  study  suggest  a  revised  definition  of 
analytical  rigor,  reframing  it  as  an  emergent  multi-attribute  measure  of  sufficiency  rather 
than  as  a  measure  of  process  deviation.  This  expanded  understanding  of  rigor  serves  as  an 
analytic  broadening  check  to  be  leveraged  against  the  risk  of  shallow  analysis. 
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INTRODUCTION 


Connectivity...  is  pervasive.  Access...  abundant.  And  more  people  now  have  available  at  their 
proverbial  fingertips  more  data,  in  more  varied  forms,  than  ever  before.  Paradoxically,  no 
matter  how  much  data  availability  technology  affords  us,  it  still  requires  a  person  to 
transform  that  data  into  information...  to  give  it  meaning  (Woods,  Patterson,  &  Roth,  2002). 

Presumably,  someone  somewhere  is  doing  just  that  at  this  very  moment — turning  data  into 
information.  These  people  are  information  analysts.  A  broad  term,  information  analyst 
describes  a  wide  array  of  people  performing  an  even  wider  array  of  tasks.  In  fact,  given  the 
access  that  the  proliferation  of  technology  affords  us,  most  all  of  us  qualify  as  information 
analysts  by  this  definition  at  one  time  or  another. 

And  indeed,  we  all  are.  From  the  middle  school  student  browsing  the  internet  on  a  home 
computer,  doing  research  for  a  book  report;  to  the  recent  graduate  thumbing  through  auto 
magazines  at  the  local  library,  preparing  for  the  purchase  of  a  new  car;  to  the  busy  parent 
contacting  family,  friends,  and  local  travel  agents,  looking  to  identify  an  affordable,  yet  fun- 
for-all-ages  destination  at  which  to  spend  an  upcoming  family  vacation — these  people  are 
unified  in  that  they  are  engaged  in  a  process  of  collecting  data  and  aggregating  it  into 
something  of  value.  This  is  the  process  of  information  analysis. 

There  is  another  important  commonality  that  spans  across  these  instances  of  analysis — and 
that  is  the  risk  of  being  too  shallow.  The  risk  of  shallow  analysis  describes  the  possibility, 
inherent  in  all  information  analysis  processes,  that  an  analysis  is  of  inadequate  depth 
relative  to  the  demands  of  the  situation — the  student,  for  example,  who  searches  only  for 
"Vikings11  and  subsequently  hands  in  a  World  History  report  asserting  that  the  Vikings  were 
from  Minnesota  (and  technically  they  are,  but  only  if  one  is  interested  in  the  National 
Football  League  team,  rather  than  the  seafaring  Scandinavians — which  the  student's  teacher 
most  probably  is  not).  The  risk,  then,  is  of  being  insufficiently  rigorous  in  analysis — a  better 
understanding  of  analytic  rigor,  in  turn,  representing  an  approach  to  cope  with  this 
potential  risk.  As  revealed  by  the  Vikings  example,  the  proliferation  of  data  availability  and 
connectivity  has  exacerbated  the  risk  of  shallowness,  as  analysis  tools  that  were  once 
confined  to  the  domain  of  experts  are  now  wielded  by  neophyte  information  analysts 
abound. 

The  expanded  access  to  data  has  impacted  more  than  the  casual  information  analyst.  This 
thesis  examines  a  different  set  of  analysts — those  who  perform  information  analysis  not  to 
pursue  personal  goals,  but  to  support  another  person  or  group  in  the  pursuit  of  their  goals. 
These  are  the  professional  information  analysts.  In  the  professional  analysis  domain,  the 
cognitive  work  of  information  analysis  and  its  complement  of  decision  making  are  often 
partitioned  across  individuals.  Thus,  the  professional  information  analyst  acts,  at  least  in 
part,  on  behalf  of  another  stakeholder — typified  as  the  decision  maker.  By  this  definition. 
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significantly  fewer  of  us  qualify  as  information  analysts,  though  some  still  do — the  broker 
checking  into  the  financial  history  of  an  investment  opportunity  on  behalf  of  investors,  the 
marketing  researcher  assembling  an  analysis  of  a  competitor's  new  products  for  a  corporate 
executive,  and  the  military  analyst  preparing  a  report  of  enemy  troop  movements  to  brief  to 
a  commanding  officer,  to  name  but  a  few. 

What  makes  professional  information  analysis  interesting  to  study  is  also  part  of  what 
makes  it  difficult — it  is  fundamentally  a  coordinated  joint  activity.  Because  multiple 
stakeholders  share  an  interest  in  the  process,  there  are  requisite  aspects  of  coordination  and 
communication  that  are  not  necessarily  required  of  other  forms  of  analysis.  It  is  from  these 
necessary  collaborative  interactions  that  many  novel  challenges  to  the  information  analysis 
process  emerge — challenges  for  which  there  are  few,  if  any,  easy  solutions.  These  challenges 
can  be  addressed,  however,  through  the  recognition  of  the  critical  connection  between  the 
professional  analysts  who  perform  information  analysis  work  and  the  decision  makers  who 
use  information  analysis  products  to  support  decision-making. 

In  addition  to  acting  on  behalf  of  another  party,  there  are  other  aspects  that  separate  these 
cases  of  professional  information  analysis  from  those  general  cases  previously  mentioned.  In 
all  instances  of  analysis,  the  risk  of  shallowness  is  fundamental — for  both  the  middle  school 
student  and  the  marketing  researcher,  to  be  shallow  is  to  chance  a  failure  of  the  analysis 
process.  However,  when  analysis  is  performed  professionally,  the  stakes  are  almost  always 
substantially  higher. 

In  the  former  information  analysis  cases,  unacceptably  shallow  analysis  leads  to  very  real 
but,  relatively  speaking,  not  very  serious  outcomes — the  student  has  to  rewrite  the  book 
report,  the  car  buyer  ends  up  with  a  lemon,  and  the  family  arrives  at  their  hotel  only  to  find 
that  it  looks  nothing  like  it  did  in  the  travel  agent's  brochure.  In  contrast,  to  fail  in 
professional  information  analysis  is  to  risk  significant  adverse  events — investors  go  bankrupt 
because  of  poor  investment  advice  or,  in  the  more  consequential  case,  soldiers  die  because 
of  inaccurate  tactical  intelligence  reports.  In  professional  analysis,  the  consequences 
resulting  from  insufficiently  rigorous  analysis  are  often  both  very  real  and  very  serious.  Thus, 
the  understanding  of  rigor  emerges  again  as  even  more  valuable  in  confronting  the  risk  of 
shallowness,  further  motivating  the  need  to  better  define  and  understand  the  concept. 

Another,  more  subtle  trait  of  professional  information  analysis  is  that,  although  seldom 
performed  exclusively  to  fulfill  a  goal  of  the  analyst,  it  is,  however,  done  with  a  specific 
purpose.  Most  often,  this  class  of  information  analysis  is  performed  with  the  purpose  of 
generating  information  to  influence  a  decision  or  action  in  the  world — the  corporate 
executive,  for  example,  may  adjust  the  company's  research  and  development  priorities  to 
keep  pace  with  a  rival  firm's  newly  released  technologies. 

The  corollary  to  this  trait,  also  a  distinct  aspect  of  professional  information  analysis,  is  that 
analysts  are  necessarily  collecting  evidence  from  a  data  space  that  is  shaped  by  other 
decision  influencing  agents.  That  is,  given  that  an  analyst  is  collecting  information  with  the 
expressed  purpose  of  influencing  a  decision  maker,  there  are  invariably  other  agents  who 
have  both  the  ability  and  the  desire  to  influence  an  analyst's  data  sources,  with  the  intent  of 
influencing  the  decision  maker  by  proxy.  In  its  mildest  form,  a  professional  analyst  must  be 
cognizant  of  the  potential  for  bias  in  data  sources.  Taken  to  an  extreme,  active  deception 
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may  be  used  by  some  agents  to  mask  adversarial  intentions  and  distort  the  data  that  is 
available  for  an  information  analyst  to  collect  and  synthesize. 

As  in  other  forms  of  research,  professional  information  analysis  also  tends  to  be  a 
hypothesis-driven  activity.  That  is,  analysts  endeavor  to  collect  and  use  data  to  generate  and 
verify — or,  perhaps  more  appropriately,  falsify — hypotheses  that  explain  the  data.  However, 
unlike  in  many  other  research  domains,  information  analysts  are  often  limited  in  their 
ability  to  control  and  vary  the  conditions  under  which  their  data  are  generated.  Both 
organizational  and  practical  factors  typically  contribute  to  limit  the  authority  and  ability  of 
analysts  to  experimentally  test  hypotheses  through  the  direct  manipulation  of  the  world  of 
interest.  Rather,  they  collect  data  that  is  most  often  generated  through  the  dynamics  of 
real-world  interactions  and  strive  to  make  sense  of  it — a  fourth  attribute  of  the  process  of 
professional  information  analysis. 

And  so  the  originally  proposed  definition  of  information  analysis  has  been  somewhat 
refined,  as  four  characteristics  were  identified  that  differentiate  professional  information 
analysis — hereinafter  referred  to  simply  as  information  analysis — from  other  more  general 
forms.  These  traits  include  the  conditions  that  information  analysis  is  (1)  performed  in 
support  of  goals  that  extend  beyond  those  of  the  analyst,  (2)  occurring  in  a  professional 
context  where  the  consequences  of  shallow  analysis  are  substantial,  (3)  done  to  influence  a 
decision  or  action  in  a  world  where  other  agents  are  also  trying  to  influence  that  same 
decision  or  action,  and  (4)  completed  with  limited  ability  to  directly  manipulate  the  world  of 
interest  in  building  and  testing  hypotheses  as  possible  explanations  for  data. 

In  this  revised  framing  of  information  analysis,  the  risk  of  shallow  analysis  is  even  greater — 
in  part  because  of  the  changing  technological  landscape  of  the  professional  analyst,  but  also 
in  part  because  the  cognitive  work  of  information  analysis  is  partitioned  across  analysts  and 
decision  makers.  Consequently,  in  information  analysis  it  is  often  difficult  for  both  analysts 
and  decision  makers  to  detect  shallow  analysis.  In  addressing  the  risk  of  shallow  analysis,  the 
understanding  of  the  concept  of  rigor  emerges  as  an  approach  for  coping  with  this 
fundamental  uncertainty.  Through  a  study  of  professional  information  analysts,  this  thesis 
examines  the  question  of  whether  providing  insight  into  an  analysis,  by  revealing  the  rigor 
of  the  analytic  process,  is  a  promising  approach  for  supporting  success  in  information 
analysis,  section 

Section  1  presents  the  background  of  the  concept  of  rigor,  exploring  how  it  is 
conventionally  defined  in  information  analysis.  Additionally,  the  section  briefly  touches 
upon  the  history  of  the  concept,  as  this  research  into  rigor  is  motivated  by  a  case  of  analysis- 
based  decision  making  where  critical  decisions  were  made  based  on  analysis  products 
perceived  as  being  rigorous,  but  that  were  in  actuality  produced  by  processes  of  very  low 
rigor.  In  this  case,  the  misperceptions  contributed  to  a  catastrophic,  loss-of-life  outcome.  This 
influential  case  frames  how  the  concept  of  rigor  is  commonly  understood  as  being 
meaningful  in  analysis  contexts. 

Section  2  discusses  the  domain  of  intelligence  analysis  as  a  natural  laboratory  for  the  study 
of  rigor  in  information  analysis.  While  there  are  many  domains  that  offer  potential  settings 
for  researching  rigor,  this  thesis  presents  findings  from  a  study  of  rigor  in  intelligence 
analysis — a  distinct,  but  arguably  representative,  instance  of  information  analysis.  This 
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section  also  identifies  the  two  basic  research  questions  that  motivate  the  study,  which  focus 
on  the  influences  of  process  insight  and  the  cues  used  in  inferring  rigor.  In  the  context  of 
this  thesis,  process  insight  refers  to  making  observable  the  analytic  process  behind  an 
analysis  product. 

Section  3  describes  a  research  study  in  which  intelligence  analysts  participated  in  an 
exploration  asking  how  professional  analysts  decide  when  there  is  sufficient  rigor  in  an 
analytic  process.  Specifically,  the  study  explored  the  role  of  process  insight  in  influencing 
perceptions  of  rigor.  In  addition,  it  explored  the  cues  that  identify  analytic  rigor,  or  lack  of 
rigor,  investigating  the  challenges  that  inhibit  the  understanding  of  rigor  in  intelligence 
analysis.  The  study  used  an  innovative  approach,  based  on  formal  knowledge  elicitation 
methodologies,  to  tap  into  the  domain  expertise  of  the  professional  intelligence  analyst. 

Nine  professional  analysts  participated  in  a  scenario  walkthrough  in  which  they  critiqued 
the  analysis  processes — one  representing  high-rigor  analysis  and  one  a  low-rigor  analysis — 
of  two  junior  analysts.  These  analysis  processes  were  developed  in  the  context  of  a 
researcher-created  information  analysis  scenario,  based  on  actual  events,  that  incorporated 
aspects  of  energy,  siting,  safety,  and  security  in  the  case.  In  the  study,  participants  assumed 
the  role  of  analyst  supervisor  facing  the  decision  of  whether  or  not  to  invest  additional 
resources  into  the  analytic  processes  of  the  junior  analysts,  by  deciding  if  the  analyses  were 
too  shallow  or  if  they  were  of  sufficient  rigor  to  send  forward  to  a  decision  maker — a 
decision  characterized  within  this  study  as  the  Supervisor's  Dilemma.  The  results  of  the 
research  study  are  also  reported  in  the  section,  presented  in  relation  to  the  prompts 
embedded  in  the  study  that  elicited  feedback  from  participants  about  how  the  concept  of 
rigor  is  embedded  in  the  domain  of  intelligence  analysis. 

Section  4  discusses  the  contributions  of  this  research  effort,  describing  their  implications  in 
relation  specifically  to  intelligence  analysis  specifically  as  well  as  to  the  broader  context  of 
information  analysis.  The  study  produced  valuable  contributions  both  in  its  methodology 
and  in  its  results.  In  method,  the  study  design  both  validated  and  refined  the  Elicitation  by 
Critiquing  approach  in  that  it  yielded  participant  insights  that  extended  beyond  surface- 
level  practitioner  observations  of  the  nature  of  their  work.  Additionally,  the  study 
developed  the  Supervisor's  Dilemma  as  a  means  for  highlighting  the  interface  between 
analysis  and  replanning  decisions,  developing  the  Liquified  Natural  Gas  (LNG)  Scenario  as  a 
cognitive  test  bed  for  exploring  the  critical  themes  of  information  analysis. 

The  results  of  the  study  identified  two  general  findings  that  serve  to  expand  the 
understanding  of  rigor  in  information  analysis.  First,  it  was  found  that  process  insight 
influences  judgments  of  rigor.  Second,  it  was  found  that  similar  cues  are  used  by  analysts  in 
forming  assessments  of  rigor,  although  the  way  in  which  those  cues  are  interpreted  as 
indicating  rigor  tend  to  be  more  varied.  The  most  surprising  finding,  however,  was  that  only 
one  of  the  nine  participants  viewed  either  scenario-based  analysis  as  being  of  sufficient 
rigor  to  send  forward.  This  finding  challenged  the  conventional  definition  of  rigor,  leading 
to  a  proposed  reconceptualization  of  a  meaningful  definition  of  rigor  in  information 
analysis.  Finally,  this  section  discusses  the  varied  perspectives  on  rigor  in  the  intelligence 
community,  identifying  future  directions  for  the  continued  exploration  of  the  concept  of 
rigor  in  information  analysis. 
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1.  ON  THE  UNDERSTANDING  OF  RIGOR 


In  information  analysis  domains,  rigor  describes  the  quality  of  an  analytic  process.  There  is, 
however,  some  debate  surrounding  what  it  means  to  be  rigorous  in  the  analysis  process.  This 
section  attempts  to  establish  a  meaningful,  operational  definition  for  analytic  rigor  within 
this  context.  Additionally,  a  brief  history  of  the  emergence  and  evolution  of  the  concept  of 
rigor  is  presented. 


1.1  Defining  Rigor 

Formally  defined,  rigor  is  the  quality  of  being  strict  and  inflexible  (Merriam-Webster's 
Collegiate  Dictionary,  2003;  McKean,  2005).  When  applied,  rigor  is  often  used  to  describe 
process.  In  information  analysis,  rigor — or  analytical  rigor — reflects  an  assessment  of  process 
quality,  affording  communication  about  the  process,  rather  than  the  product,  of  analysis. 
One  cooperative  effort  to  identify  a  generic  definition  of  analytical  rigor  describes  it  as  the 
"application  of  precise  and  exacting  standards...  to  better  understand  and  draw  conclusions...  based 
on  careful  consideration  or  investigation"  (Military  Operations  Research  Society,  2006).  In  this 
view,  to  be  rigorous  in  analysis  requires  only  the  meticulous  adherence  to  standard  process. 

This  colloquial  perspective,  however,  mischaracterizes  the  understanding  of  rigor  in 
information  analysis.  Despite  the  etymological  implication  that  to  be  rigorous  is  to  "be  stiff" 
expert  information  analysis  processes  often  are  not  rigid  in  their  application  of  a  standard 
process,  but  rather,  flexible  and  adaptive  to  highly  dynamic  environments  (Merriam- 
Webster's  Collegiate  Dictionary,  2003;  McKean,  2005).  In  information  analysis,  judgement  of 
rigor  reflects  a  relationship  in  the  appropriateness  of  fit  between  analytic  processes  and 
contextual  requirements.  Thus,  as  supported  by  the  research  presented  in  this  thesis,  rigor  is 
more  meaningfully  viewed  as  an  assessment  of  degree  of  sufficiency,  rather  than  degree  of 
adherence  to  an  established  analytic  procedure. 

This  debate  about  what  it  means  to  be  rigorous  in  analysis  does  not  lessen  the  important 
role  of  understanding  rigor.  Rather,  it  shapes  the  perspective  taken  in  exploring  the 
understanding  of  rigor  in  information  analysis. 


1.2  A  Brief  History  of  Rigor 

The  research  on  rigor  in  information  analysis  began  in  engineering  safety  analysis,  where 
significant  and  high-profile  "failures"  made  salient  the  criticality  of  understanding  analytic 
rigor.  At  NASA,  following  the  Columbia  accident,  post-accident  investigation  reports 
revealed  that  managers  were  unaware  they  were  making  decisions  based  on  analyses  that 
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appeared  thorough,  but  were  in  fact  of  very  low  rigor  (Columbia  Accident  Investigation 
Board,  2003;  Crippen,  et  al.,  2005;  Woods,  2005). 

Prior  to  the  foam  strike  event  that  ultimately  fated  the  STS-1 07  mission,  some  NASA 
engineers  and  managers  were  concerned  about  the  uncertain  risk  associated  with  the 
recognized  problem  of  foam  loss  from  the  shuttle's  External  Tank.  However,  the 
understanding  of  this  problem  was  not  validated  by  rigorous  engineering  analyses,  and  past 
successes  became  the  basis  for  justifying  future  flights.  As  described  in  the  Columbia 
Accident  Investigation  Board  (2003)  report: 

With  each  successful  landing,  it  appears  that  NASA  engineers  and  managers 
increasingly  regarded  the  foam-shedding  as  inevitable,  and  as  either  unlikely  to 
jeopardize  safety  or  simply  an  acceptable  risk. ..NASA  and  contractor  personnel 
came  to  view  foam  strikes  not  as  a  safety  of  flight  issue,  but  rather  a  simple 
maintenance,  or  "turnaround"  issue... 

What  was  originally  considered  a  serious  threat  to  the  Orbiter  came  to  be  treated 
as....  a  reportable  problem  that  was  within  the  known  experience  base,  was 
believed  to  be  understood,  and  was  not  regarded  as  a  saf ety-of-f I ig ht  issue... 

[However,]  assessments  of  foam-shedding  and  strikes  were  not  thoroughly 
substantiated  by  engineering  analysis. 

Moreover,  as  reports  addressing  the  foam  loss  events  progressed  through  the  organization, 
decision  makers  increasingly  failed  to  realize — in  light  of  increasing  pressures  to  remain  on 
schedule — that  the  processes  behind  these  analyses,  and  thus  the  bases  for  their  decisions, 
were  insufficiently  rigorous  relative  to  the  safety-critical  questions  being  asked  and  the 
potential  consequences  of  the  decisions  being  made. 

It  is  in  this  context  that  a  definition  of  rigor  as  "the  scrupulous  adherence  to  established 
standards  for  the  conduct  of  work"  emerges  (Crippen,  et  al.,  2005).  However,  as  noted  earlier, 
this  definition  of  rigor  is  also  insufficient  in  much  the  same  way  that  an  error-as-deviation 
from  standards  definition  of  "human  error"  in  safety  analysis  is  inadequate:  "defining  error- 
as-deviation  from  a  model  of  'good'  process...  collides  with  the  problem  of  multiple 
standards"  (Woods  &  Cook,  2003).  The  "problem  of  multiple  standards"  describes  the 
inherent  difficulties  in  selecting  which,  among  many  viable  candidates,  is  the  standard 
process  to  which  performance  should  be  compared.  In  information  analysis — intelligence 
analysis,  especially — it  is  pragmatically  even  more  challenging  to  define  standard  process. 

The  standards-based  perspective  on  rigor  is  also  insufficient  in  that  it  fails  to  consider  the 
influence  of  time  pressure  on  analytic  rigor.  As  dictated  by  the  speed-accuracy  trade-off,  the 
amount  of  time  available  to  perform  an  analysis  will  have  a  direct  impact  on  its  quality 
(Wickens  &  Hollands,  2000).  Thus,  assessing  the  rigor  of  an  analysis  product  requires  some 
consideration  of  the  constraints  under  which  it  was  produced.  This  finding,  in  conjunction 
with  the  recognized  importance  of  understanding  the  level  of  rigor  demanded  for  a 
particular  decision,  reframes  rigor  more  meaningfully  as  a  measure  of  sufficiency,  rather 
than  as  a  measure  of  deviation. 

Engineering  safety  analysis  is  not  the  only  domain  where  understanding  the  role  of 
judgments  about  the  rigor  of  analyses  is  recognized  as  important.  In  the  physical  sciences, 
the  significance  of  rigor  is  embodied  in  the  canonical  "scientific  method"  of 
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experimentation.  As  noted  by  Davies  and  Dodd  (2002),  "rigor  is  the  authoritative  evaluation  of 
good  research  and  the  unspoken  standard  by  which  all  research  is  measured."  In  the  social  sciences 
— although  the  definition  of  what  constitutes  rigor  is  a  bit  murkier — the  appreciation  of  the 
importance  of  rigorous  process  remains. 

It  is  also  recognized,  in  various  manifestations,  that  poor  representation  hides  and  obscures 
the  process  being  represented.  Thus,  rigor  represents  a  mechanism  for  revealing  the 
observability  of  an  analysis  process  (Woods,  Patterson,  &  Roth,  2002).  In  the  design 
literature,  Tufte  (2003)  is  especially  critical  of  the  influence  of  slide-based  presentations  on 
the  understanding  of  analytical  process.  Microsoft  PowerPoint  in  particular,  he  argues, 
encourages  the  acceptance  of  unacceptably  low-rigor  analyses.  At  the  root  of  his  criticism  is 
this  same  observation — that  the  form  of  an  analysis  product  can  hide  the  process  behind  it. 
This  finding  is  extended  through  the  acknowledgment  that  all  forms  of  analytic  products,  to 
some  extent,  obscure  the  processes  that  produced  them — an  insightful  perspective  that 
guides  the  exploration  of  rigor  in  information  analysis. 
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2.  RESEARCHING  RIGOR  IN  INFORMATION  ANALYSIS 


This  work  serves  to  expand  the  understanding  of  rigor  in  information  analysis  by  studying  it 
in  the  domain  of  professional  intelligence  analysis.  This  section  describes  the  context  of 
intelligence  analysis  as  a  natural  laboratory  for  studying  the  concept  of  rigor.  Additionally, 
this  section  identifies  the  specific  research  questions  that  guided  the  exploration  of  rigor. 


2.1  Intelligence  Analysis  as  Information  Analysis 

In  the  introduction,  information  analysis — as  a  manifestation  of  abductive  reasoning 
(Josephson  &  Josephson,  1994) — is  defined  loosely  as  a  form  of  cognitive  work  that  involves 
the  collection  and  assessment  of  data  with  the  goal  of  verifying  or  refuting  hypotheses 
which  explain  that  data  (Elm,  et.  al.,  2005).  Information  analysis  is  more  than  just 
explanation  building  and  testing,  however,  as  it  is  performed  to  support  high-consequence 
decision-making  in  domains  where  the  events  to  be  explained  or  the  projections  of  how 
events  will  develop  may  not  be  testable  in  the  same  way  as  one  would  in  classic 
experiments.  Intelligence  analysis — in  particular  strategic,  rather  than  tactical  or  operational 
intelligence — provides  a  representative  natural  laboratory  for  refining  the  understanding  of 
rigor  in  information  analysis. 

Intelligence  analysis,  as  a  form  of  abduction  (Schum,  1987),  is  a  special  case  of  information 
analysis.  While  it  shares  the  previously  noted  features  with  most  other  information  analysis, 
it  is  confounded  by  the  added  challenges  of  necessary  secrecy  and  adversarial  intent — as 
decision  influencing  agents  operating  within  this  context  employ  deception  as  the  extreme 
form  of  decision  influencing  factor.  Similar  to  the  above  definition  of  information  analysis, 
Johnson  (2005)  defines  intelligence  analysis  as  "the  application  of  individual  and  collective 
cognitive  methods  to  weigh  data  and  test  hypotheses  within  a  secret  socio-cultural  context."  This 
characterization  partially  overlaps  with  the  model  proposed  by  Elm,  et  al.  (2005),  which 
frames  intelligence  analysis  as  an  interaction  of  the  three  primary  functions  of  Down  Collect 
of  relevant  data,  construction  of  interpretations  of  the  data  through  Conflict  and 
Corroboration,  and  building  explanations  for  those  interpretations  through  Hypothesis 
Exploration. 

In  the  interest  of  clarity,  it  should  be  noted  that  there  are  three  senses  in  which  the  term 
"intelligence  analysis"  or,  more  simply,  "intelligence"  is  colloquially  used  (Kent,  1966).  Most 
commonly,  intelligence  analysis  describes  the  process  of  analysis.  However,  the  term  can  also 
be  used  in  reference  to  the  product  of  an  analysis  process,  for  example,  in  reference  to  an 
intelligence  briefing  report.  By  this  definition,  an  intelligence  analysis  processes  produce 
intelligence  analysis  products.  Less  frequently,  intelligence  analysis  describes  the  broader 
analysis  process  of  the  intelligence  community — a  cycle  traditionally  including  collection, 
processing,  analysis,  dissemination,  and  planning  (Doctrine  for  Intelligence  Support  to  Joint 
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Operations,  2000;  Johnson,  2005).  In  this  framing,  intelligence  analysis  is  a  component  of  the 
intelligence  cycle. 


In  the  context  of  this  work,  every  effort  is  made  to  use  the  term  "intelligence  analysis"  only 
the  first  sense — in  reference  to  the  actual  process  of  analysis.  Consequently,  intelligence 
analysis  products  are  referred  to  as  intelligence  reports  or  intelligence  briefings.  And,  in 
keeping  with  convention,  broader  analysis-related  interactions  within  the  intelligence 
community  are  referred  to  as  the  intelligence  cycle,  rather  than  as  intelligence  analysis. 

Intelligence  analysis  is  a  particularly  interesting  domain  for  studying  rigor  because,  in  view 
of  the  consequences  of  failure,  recent  high-profile  events  have  prompted  many  to  question 
the  present  state  of  the  U.S.  intelligence  community  (National  Commission  on  Terrorist 
Attacks,  2004;  Duelfer,  2004;  Hammond,  2004;  Johnson,  2005).  In  the  ensuing  debates,  it  has 
become  apparent  that  changes  in  technology  and  in  the  international  political  landscape, 
among  other  influences,  have  transformed — and  continue  to  transform — the  role  of  the 
professional  analyst  (Medina,  2002;  Ward,  2002).  In  response,  there  is  a  renewed  effort  to 
better  understand  the  factors  that  impact  the  performance  of  intelligence  analysts. 

Table  1  shows  a  number  of  different  perspectives  in  the  intelligence  literature  aimed  at 
characterizing  the  challenges  and  vulnerabilities  that  influence  the  performance  of 
intelligence  analysts.  Note  that,  although  the  order  in  which  the  challenges  are  listed  in 
each  row  is  somewhat  arbitrary,  the  order  of  the  columns  are  not.  On  one  end  of  the 
spectrum,  researchers  such  as  Krizan  (1999)  and  Heuer  (1999)  focus  on  identifying  the 
misperceptions  and  biases  that  degrade  the  performance  of  the  individual  analyst,  while 
some,  such  as  Johnson  (2005),  focus  on  understanding  the  situational  and  organizational 
factors  that  hinder  the  collective  performance  of  professional  analysts.  Others  have  taken  a 
more  moderate  position,  recognizing  that  both  types  of  factors  play  a  role  in  determining 
analytic  performance  (Hutchins,  Pirolli,  &  Card,  2006;  Trent,  Patterson,  &  Woods,  in  press). 

Another  perspective  in  addressing  the  challenges  of  intelligence  analysis  focuses  not  on  the 
cognitive  biases  or  situational  attributes  that  impact  behavior,  but  on  the  interactions 
among  agents  who  support  the  intelligence  analysis  process — most  notably,  the  interaction 
between  analysts  and  decision  makers.  In  the  face  of  significant  production  pressures 
(Johnson,  2005)  and  rapidly  proliferating  data  availability  (Patterson,  Roth,  &  Woods,  2001) 
— and  resulting  data  overload  deluging  the  professional  analyst — it  is  increasingly  easy  for 
analysts  to  be  trapped  by  shallow,  low-rigor  analysis.  Given  similar  pressures,  it  is  also 
increasingly  difficult  for  decision  makers  to  recognize  when  an  analysis  is  not  of  sufficient 
rigor  for  a  given  decision.  Thus,  it  is  increasingly  clear  that  analytical  rigor  is  a  relevant 
concept  within  the  intelligence  analysis  community,  contributing  to  its  appropriateness  as  a 
valid  context  for  exploring  the  broader  theme  of  rigor  in  information  analysis.  In  the 
context  of  this  work,  the  exploration  of  the  concept  of  rigor  is  guided  by  the  specific 
research  questions  defined  in  the  following  section. 
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Krizan3 

Heuerb 

Trent,  et  al.c 

Hutchins,  et  al.d 

Johnsone 

Prematurely  Formed 
Views 

The  Vividness 
Criterion 

Inappropriate 
Mental  Set 

High  Cognitive 
Workload 

Secrecy  versus 
Efficacy  Tradeoff 

Willful  Disregard  of 
New  Evidence 

Absence  of 

Evidence 

Environmental 

Pressure 

Potential  for 

Error 

Focus  on  Current 
Production 

Lack  of  Empathy 

Base-Rate  Fallacy 

Fixation 

Time  Pressure 

Time  Constraints 

Ethnocentrism  & 
Mirror-Imaging 

Oversensitivity  to 
Consistency 

Recognition  of 
Relevant  Data 

Coping  with 
Uncertainty 

Confirmation  Bias, 
Norms,  and  Taboos 

Ignorance 

Anchoring 

Trust 

Data  Overload 

Analytic  Identity 

Rational-Actor 
Hypothesis  or  Denial 
of  Rationality 

Assessing 

Probability  of  a 
Scenario 

Experience 
viewed  as 
Expertise 

Synthesizing 
Multiple  Sources 
of  Information 

Production-based 
Rewards  and 
Incentives 

Proportionality  Bias 

Availability  Rule 

Learning 

Insufficient  Tools 

Analytic  Training 

Defensive  Avoidance 
&  Wishful  Thinking 

Conservatism  in 
Probability  Estimation 

Similarity  of  Cause 
and  Effect 

Internal  vs.  External 
Causes  of  Behavior 

Tool 

Understanding 

Sustained 

Attention 

Organizational 

Context 

Complex  Human 
Judgments 

Perception  of 
"Tradecraft" 

Versus  Scientific 
Methodology 

Presumption  that 
Support  for  One 
Hypothesis 

Disconfirms  Others 

Persistence  of 
Impressions  Based 
on  Discredited 
Evidence 

Best-Case  Analysis  or 
Worst-Case  Analysis 

Overestimating  Our 
Own  Importance 

Image  and  Self-Image 

Illusory  Correlation 

Overconfidence  in 
Subjective  Estimates 

Expression  of 
Uncertainty 

Inappropriate 
Analogies  & 

Superficial  Lessons 
from  History 

Bias  Favoring 
Perception  of 
Centralized 

Direction 

Evoked-Set  Reasoning 

Excessive  Secrecy 

Coping  with 

Evidence  of 
Uncertain  Accuracy 

Presumption  of 

Unitary  Action  by 
Organizations  & 
Organizational 
Parochialism 

Bias  in  Favor  of 
Causal  Explanations 

Note.  aAdapted  from  Intelligence  Essentials  for  Everyone  by  L.  Krizan,  1999.  bAdapted  from  Psychology  of 
Intelligence  Analysis,  by  R.  J.  Heuer,  1999.  cAdapted  form  "Challenges  for  Cognition  in  Intelligence  Analysis,"  S. 
A.  Trent,  et  al.,  in  press,  Journal  of  Cognitive  Engineering  and  Decision  Making.  dAdapted  from  "What  Makes 
Intelligence  Analysis  Difficult?  A  Cognitive  Task  Analysis  of  Intelligence  Analysts,"  S.  G.  Hutchins,  et  al.,  2006,  in 
P.  L.  Pirolli  (Ed.)  Assisting  People  to  Become  Independent  Learners  in  the  Analysis  of  Intelligence.  eAdapted  from 
Analytic  Culture  in  the  US  Intelligence  Community:  An  Ethnographic  Study,  by  R.  Johnson,  2005. 


Table  7.  Intelligence  Analysis  Challenges  and  Vulnerabilities 
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2.2 


Research  Question 


The  overarching  purpose  of  the  research  presented  in  this  thesis  is  to  refine  the 
understanding  of  rigor  by  exploring  the  role  that  the  concept  of  rigor  plays  in  information 
analysis.  As  noted  previously,  a  better  understanding  of  rigor  can  be  leveraged  against  the 
ever-present  risk  of  being  too  shallow  in  information  analysis. 

The  primary  research  question  addressed  by  this  thesis  is:  How  does  insight  into  the  analysis 
process  influence  judgement  of  rigor?  Additionally,  this  thesis  investigates  the  related, 

secondary  research  question:  What  cues  do  analysts  use  in  making  assessments  of  rigor? 

These  research  questions  are  explored  through  a  study  of  rigor  in  professional  intelligence 
analysis,  which  provided  the  context  for  the  study  of  rigor  in  information  analysis  more 
broadly.  This  research  study,  discussed  in  the  next  section,  yielded  valuable  contributions 
both  in  method  and  in  findings. 
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3.  A  STUDY  OF  THE  PROFESSIONAL  INTELLIGENCE  ANALYST 


To  address  the  primary  research  questions,  a  study  of  the  professional  analyst  was 
undertaken  to  explore  how  the  concept  of  rigor  resonated  within  the  domain  of 
intelligence  analysis.  Rather  than  resolving  the  question  of  precisely  what  does  or  does  not 
qualify  as  rigorous  analysis,  this  research  studied  how  rigor  is  understood  within  the 
intelligence  analysis  domain.  In  developing  the  study  method,  the  LNG  Scenario  defined  the 
context  while  the  Supervisor's  Dilemma  shaped  the  methodology.  The  first  four  sections  of 
this  section  discuss  the  method  of  study  and  include  a  description  of  the  participants,  the 
development  of  the  study  LNG  Scenario,  the  background  of  the  Supervisor's  Dilemma,  and  the 
scenario  walkthrough  procedure.  The  final  two  sections  discuss  the  results  of  the  study, 
describing  how  the  data  were  analyzed  and  identifying  the  major  findings  that  emerged 
from  the  research. 


3.1  Participants 

Nine  professional  intelligence  analysts  (P0-P8)  participated  in  the  study,  consisting  of  eight 
primary  participants  and  one  pilot  participant.  The  pilot  participant  (PO)  was  distinct  from 
the  primary  participants  (P1-P8)  in  that  he  represented  a  different  intelligence  organization 
than  the  other  eight.  Additionally,  minor  adjustments  were  made  to  the  verbal  protocol 
used  in  the  study  based  on  feedback  from  this  participant.  However,  as  no  significant 
changes  were  made  to  the  study — and  all  core  components  remaining  constant  across  all 
participants — the  results  of  the  pilot  study  participant  are  not  treated  as  distinct  from  those 
of  the  other  participants. 

The  analysts  selected  for  the  study  were  volunteers  who  responded  to  an  IRB-approved,  e- 
mail  recruitment  request  for  participants  made  via  a  contact  at  a  cooperating  intelligence 
organization.  All  analysts  who  volunteered  were  accepted,  although  one  potential  study 
participant  was  turned  away  due  to  scheduling  confusion  which  resulted  in  two  analysts 
arriving  for  the  same  study  session.  This  analyst  was  unable  to  be  rescheduled.  No 
compensation  was  provided  for  participation  in  this  research. 

For  reasons  of  confidentiality,  limited  personal  and  demographic  information  was  collected 
from  study  participants.  Formally,  only  two  types  of  information  were  requested:  years  of 
experience  and  familiarity  with  the  scenario  topic.  The  study  included  analysts  with  a  broad 
range  of  professional  experience  ( Median  =  20  years.  Mean  =  21  years,  SD  =  10  years),  from 
junior  analyst  with  less  than  three  years  of  experience  to  senior  analyst  with  over  forty  years 
of  experience.  Note  that  one  novice  analyst  was  recruited  to  participate  in  the  study  in 
order  to  provide  a  mechanism  for  gauging  the  role  of  domain  expertise  in  assessing  rigor. 
Regarding  familiarity  with  the  scenario,  formally  defined  in  the  next  section,  none  of  the 
participants  indicated  that  they  had  any  notable  prior  professional  or  personal  experience 
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with  the  topic.  As  per  the  IRB-approved  procedure,  only  verbal  consent  of  participation  was 
collected  from  participants  in  order  to  ensure  protection  of  their  identities. 

Informally,  some  descriptive  information  about  participants  was  revealed  during  the 
individual  research  sessions.  Most  notably,  all  of  the  professional  analysts  who  participated 
in  the  study  represented  the  strategic,  in  contrast  with  the  tactical,  intelligence  community. 
However,  the  participants  varied  with  respect  to  their  intelligence  backgrounds  (e.g., 
economics,  linguistics,  aeronautical  engineering,  etc.)  and  current  principle  duties  (e.g., 
senior  analyst,  linguist,  analyst  supervisor,  etc.).  The  study  participants  also  varied  with 
respect  to  gender  (6  male,  3  female). 


3.2  LNG  Scenario 

The  research  study  was  based  on  a  scenario  that  defined  the  context  of  the  interactions 
between  participants  and  researchers.  Study  participants,  from  the  perspective  of  an  analyst 
supervisor,  critiqued  the  analysis  processes  of  two  junior  analysts  in  the  context  of  a  defined 
scenario.  This  section  of  the  thesis  provides  a  description  of  that  scenario,  a  summary  of  how 
the  scenario  was  adapted  for  the  study,  and  information  about  the  selection  of  the  scenario 
as  well  as  about  its  design  process. 

In  order  to  preclude  a  confusion  of  terms,  throughout  the  section  the  hypothetical  junior 
analysts,  who  are  the  focus  of  the  scenario,  will  hereinafter  be  referred  to  as  "analysts"  or 
"junior  analysts",  while  the  real  professional  intelligence  analysts  who  participated  in  the 
study  will  be  referred  to  as  "participants"  or  "participant  analysts".  With  regard  to  the 
scenario,  the  participant  analysts  might  also  be  referenced  in  their  assumed  role  as 
"supervisor"  of  the  junior  analysts. 

3.2. 1  Description 

The  Liquified  Natural  Gas  (LNG)  Scenario  reflects  the  debate  in  the  United  States  energy  sector 
as  to  whether  the  importation  of  LNG  is  an  appropriate  approach  for  coping  with  increasing 
natural  gas  consumption,  in  light  of  the  projected  near-term  depletion  of  the  domestic 
natural  gas  supply.  In  the  context  of  this  debate,  to  significantly  increase  U.S.  natural  gas 
imports  would  necessitate  the  construction  of  numerous  LNG  terminals — which  often  serve 
as  the  primary,  and  most  visible,  focus  of  the  controversies  surrounding  the  importation  of 
natural  gas.  While  there  are  many  details  related  to  LNG  importation  that  make  it  an 
interesting  base  scenario,  there  are  four  facets  of  the  controversy — energy,  siting,  safety, 
and  security — that  interact  to  make  it  particularly  valuable  as  a  cognitive  case.  In  fact,  the 
scenario  developed  for  this  study  serves  as  a  case  that  is  useful  not  only  for  exploring  the 
understanding  of  rigor  in  information  analysis,  but  also,  potentially,  for  future  explorations 
of  other  aspects  of  information  analysis  as  well. 

At  the  broadest  level,  the  issue  of  LNG  importation  stems  from  disagreements  about  energy 
policy.  As  in  many  energy  policy  cases,  central  to  the  debate  is  the  tradeoff  between  the 
continued  short-term  exploitation  of  a  finite  natural  resource  and  the  long-term  need  to 
develop  viable  energy  alternatives.  Related  are  the  economic  and  geopolitical  concerns  of 
further  increasing  U.S.  dependence  on  foreign  energy  suppliers,  as  in  the  somewhat 
analogous  case  of  crude  oil  importation.  Also  factoring  into  this  decision  are  the  tradeoffs 
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between  the  economic  imperatives  driving  increasing  consumption  and  the  potential 
adverse  environmental  impacts  arising  from  the  industrialization  necessary  to  increase 
natural  gas  importation.  Although  this  list  of  tradeoffs  is  by  no  means  exhaustive,  at  a 
cognitive  level  the  energy  tradeoffs  in  the  LNG  Scenario  are  fundamentally  about 
replanning.  Here,  replanning  describes  the  need  to  regularly  reassess  and  re-plan  natural 
gas  energy  policy  to  respond  to  the  changing  consumption  of  a  limited  resource  in  a 
dynamic  energy  market. 

The  next  critical  facet  of  the  LNG  Scenario  is  the  debate  over  terminal  siting.  Much  like  the 
issue  of  energy  policy,  tradeoffs  are  at  the  core  of  this  issue — in  this  case,  the  those  tradeoffs 
arising  during  determinations  of  where  LNG  receiving  terminals  should  be  located.  As  in  the 
parallel  cases  of  landfill  and  prison  sitings,  there  is  often  local  opposition  to  new 
construction,  even  when  the  more  global  benefits  of  development  are  valued.  This 
occurrence — sometimes  described  as  the  "not  in  my  backyard"  (NIMBY)  phenomenon — 
introduces  the  concept  of  factions  into  the  LNG  Scenario,  as  different  stakeholders  accrue 
differing  levels  of  benefit  relative  to  the  assumed  risk  of  a  specific  terminal  siting.  From  a 
cognitive  perspective,  this  aspect  of  the  scenario  represents  a  more  basic  type  of  planning 
decision.  In  contrast  with  energy  policy  planning,  which  is  inherently  recurrent — and  thus 
represents  replanning — LNG  terminal  siting  is  typically  an  all-or-none  proposition,  each 
instance  of  which  represents  a  largely  unique  planning  decision.  Siting  decisions,  then,  are 
manifested  in  the  LNG  Scenario  in  the  weighing  of  tradeoffs  relative  to  the  local  and  global 
goals  that  conflict  in  planning  the  expanded  infrastructure  necessary  for  LNG  importation. 

The  third  aspect  of  the  LNG  importation  controversy  identified  as  being  a  critical  component 
of  the  overall  scenario  is  the  issue  of  safety.  Safety  issues  encompass  many  facets  of  the 
liquified  natural  gas  importation  process,  though  in  the  U.S.  these  concerns  most  often 
focus  on  the  tankers  that  transport  LNG  and  the  receiving  terminals  that  offload  it  from  the 
tankers.  In  both  cases,  safety  is  a  particularly  relevant  issue  because  there  is  little  that  is 
definitively  known  about  the  impacts  of  a  large-scale  release  of  LNG,  as  few  safety  analyses 
address  this  issue  specifically.  Moreover,  in  light  of  the  limited  research  base,  there  is  some 
concern  that,  as  a  volatile,  compressed  gas,  the  worse-case  outcome  of  a  widespread  release 
could  reach  catastrophic  proportions.  Thus,  while  the  benefits  are  fairly  well  defined,  the 
nature  of  the  risks  remains  largely  unverified.  As  a  cognitive  aspect  of  the  study,  these  safety 
issues  can  be  thought  of  as  risk  versus  benefit  tradeoff  analysis.  In  addition,  as  the  LNG 
safety  assessments  that  do  exist  are  themselves  information  analysis  products,  the  scenario 
also  has  an  embedded  component  of  nesting  in  that  it  is  effectively  an  analysis-of-analysis. 

The  final  dimension  of  the  controversy — security  concerns — is  an  especially  interesting 
aspect  of  the  LNG  Scenario  because  it  reflects  a  form  of  safety  tradeoff  that  was,  at  one  time, 
not  viewed  as  being  significantly  different  from  other  safety  concerns.  While  traditional 
safety  analysis  primarily  focused  on  accidental  LNG  releases — such  as  those  attributable  to 
"human  error"  or  natural  disasters — the  recognized  risk  of  terrorism  led  to  the  rethinking  of 
previous  safety  analyses  with  a  specific  consideration  of  the  threat  of  an  intentional  LNG 
release.  Consequently,  in  light  of  the  fact  that  this  critical  failure  mode  was  not  adequately 
considered  in  prior  analyses,  many  of  the  conclusions  and  assumptions  about  the  inherent 
safety  of  LNG  importation  were  challenged,  leading  to  revised  assessments  of  risk  and  safety 
tradeoffs.  The  change  in  security  concerns,  then,  served  as  a  catalyst  for  reassessment.  At  the 
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cognitive  level,  this  aspect  of  the  LNG  Scenario  reflects  an  event-driven  reanalysis  of 
previously  established  perceptions  of  safety. 

The  four  basic  aspects  of  energy,  siting,  safety,  and  security  define  the  importation  of  LNG  as 
a  multidimensional  case  for  studying  the  concept  of  rigor  in  information  analysis.  From  the 
cognitive  perspective,  the  development  of  this  scenario  is  particularly  valuable  in  that  it  taps 
into  both  the  analysis  and  planning  aspects  of  information  analysis — through  LNG  safety 
analysis  and  site  planning  issues,  respectively.  Additionally,  the  scenario  captures  a  setting 
that  is  significantly  influenced  by  the  dynamics  of  a  changing  world  through  the  functions 
of  reanalysis  and  replanning — reflected  by  the  event-driven  security  reanalysis  and  future- 
oriented  energy  policy  planning.  Taken  together,  these  four  cognitive  components  of  the 
LNG  Scenario — analysis  and  planning,  reanalysis  and  replanning — serve  as  basic  cognitive 
levers,  built  into  the  scenario,  that  can  be  manipulated  to  understand  varied  themes  in 
information  analysis. 

3.2.2  Adaptation 

In  the  scenario,  two  junior  intelligence  analysts — Analyst  1  and  Analyst  2 — were  tasked  by  a 
decision  maker  to  answer  a  question  related  to  LNG  as  a  factor  in  a  U.S.  energy  policy 
decision.  The  specific  task  question  driving  the  scenario  was,  "What  are  the  primary  obstacles 
to  using  Liquified  Natural  Gas  to  address  energy  needs  in  the  U.S.?"  (Appendix  A,  Figure  A.1). 

Each  junior  analyst  performed  an  analysis  and  prepared  a  briefing  report  addressing  this 
task  question.  The  responses  of  Analyst  1  and  Analyst  2  provided  the  processes — Analysis 
Process  1  [A(1)j  and  Analysis  Process  2  [A(2)j,  respectively — that  were  critiqued  in  the  study. 

The  request  by  the  hypothetical  decision  maker  in  the  scenario  was  framed  as  a  quick 
reaction  task,  meaning  that  the  junior  analysts  each  spent  a  few  hours  over  the  course  of  a 
couple  days  producing  a  response  to  the  task  question.  Each  analyst  spent  a  comparable 
amount  of  time  responding  to  the  task  question.  The  backgrounds  of  the  analysts  were  also 
similar,  so  as  not  to  be  a  significant  factor  in  the  scenario.  Throughout  the  study, 
participants  assumed  the  role  of  the  hypothetical  supervisor  of  these  junior  analysts, 
responsible  for  deciding  if  their  analysis  briefing  reports  were  of  acceptable  quality  to 
forward  to  the  decision  maker  who  initiated  the  request — thus  engaging  the 
Supervisor's  Dilemma. 

3.2.3  Selection 

The  Liquified  Natural  Gas  Scenario  was  selected  from  among  six  potential  scenario  topics,  listed 
in  Table  2,  that  emerged  from  iterative  brainstorming  sessions  between  investigators  and 
intelligence  analysis  practitioners.  The  primary  determinants  in  choosing  a  scenario  were,  in 
approximate  order  of  importance:  authenticity,  classification  status  of  information,  scope, 
and  topical  interest.  While  no  formal  scoring  methodology  was  employed  in  comparing  and 
contrasting  the  scenarios,  they  were  weighed  and  considered  in  consultation  with  two 
professional  analysts.  The  insights  and  perspectives  offered  by  these  analysts  contributed  to 
the  selection  of  LNG  as  the  final  study  scenario  topic.  The  two  analysts  were  from  the 
strategic  intelligence  community  and  represented  the  same  intelligence  organization  as  the 
pilot  study  participant. 
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Domain 

Topic 

Focus 

Energy 

U.S.  Importation  of  Liquid 
Natural  Gas  (LNG) 

U.S.  natural  gas  infrastructure. 

World  Health 

Pandemic  Flu  Outbreak 
(e.g.  Spread  of  Avian  Flu) 

Impact  on  U.S.  and  international 
transportation  and  travel. 

International  Finance 

Collapse  of  the  Euro 

Likelihood  of  occurrence  and  impact  on 
sustainability  of  the  E.U. 

Energy  /  International 
Relations 

Russia  /  Ukraine  Natural  Gas 
Pricing  Dispute 

Future  governmental  involvement  in 

Russian  energy  supplier  (Gazprom) 
decisions  and  policies. 

Energy 

E.U.  Transition  to  Hydrogen- 
Based  Economy 

Required  changes  for  initial  transition. 

Technology 

International  Copyright  Issues 

Strain  on  U.S.  /  China  relations  resulting 
from  internal  pressures  for  enforcement. 

Table  2.  Potential  Scenario  Topics  Considered 

In  scenario-based  studies  it  is  particularly  important  that  the  scenario  be  viewed  as  authentic 
by  participants,  as  it  influences  the  degree  to  which  they  view  a  study  as  meaningful  and 
relevant.  Consequently,  it  influences  the  extent  to  which  they  are  willing  to  engage  in  a 
scenario — which  relates  directly  to  the  quality  of  feedback  they  provide.  In  that  light,  it 
should  be  noted  that  the  LNG  Scenario  was  chosen  not  because  of  its  authenticity  as  a 
strategic  intelligence  analysis  task,  but  rather  because  of  its  authenticity  as  a  representative 
information  analysis  task.  Thus,  it  was  intentional  that  the  scenario  chosen  for  the  study 
that  was  not  exactly  like  the  work  of  the  professional  intelligence  analysts  who  participated 
in  the  study. 

A  highly  authentic,  intelligence-specific  task  was  not  selected  for  two  primary  reasons.  First, 
the  scenario  was  chosen  to  avoid  topics  of  direct  relevance  to  any  participant.  Because  the 
focus  of  the  study  was  on  process,  rather  than  on  whether  the  conclusions  conveyed  in  the 
reports  were  right,  the  topic  selected  was  intended  to  be  outside  the  specific  areas  of 
expertise  of  the  participants.  This  avoided  the  possibility  that  a  participant  might  have  a 
predisposition  toward  a  scenario  that  would  restrict  their  ability  to  provide  unbiased 
feedback.  As  noted  previously,  the  selection  of  the  scenario  topic  was  at  least  informally 
successful  in  this  respect,  as  none  of  the  participants  indicated  any  significant  experience 
with  the  topic  of  LNG. 

Second,  not  selecting  a  heavily  intelligence-focused  scenario  avoided  introducing  the 
problem  of  participants  revealing  classified  information.  Because  of  the  inherently  sensitive 
nature  of  the  work  of  intelligence  analysts,  the  scenario  topic  selected  was  intended  to 
allow  participants  to  speak  openly.  Selecting  a  authentic  intelligence  topic  risked  restricting 
the  ability  of  participants  to  communicate  their  perspectives  candidly  throughout  the  study. 
Moreover,  it  risked  being  a  distraction  from  the  focus  of  the  study,  which  was  to  explore  the 
understanding  of  analytic  processes,  rather  than  products. 

In  choosing  a  scenario  it  was  also  important  that  all  process  inputs  came  from  unclassified 
sources  in  order  to  address  the  issue  of  scope,  the  third  selection  criteria  used.  As  there  was 
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a  limited  amount  of  time  available  for  each  interview,  relying  only  on  open  source  data 
aided  in  developing  a  tractable  scenario  case.  It  was  determined  that  the  LNG  Scenario  was  a 
topic  of  acceptable  scale,  given  the  pragmatic  constraints  of  running  the  study. 

The  final  consideration  in  selecting  the  scenario  was  its  anticipated  level  of  topical  interest. 

It  was  determined  that  the  importation  of  LNG  is  a  relevant  issue  in  many  areas  of  the 
United  States,  in  light  of  the  current  status  of  U.S.  energy  needs  relative  to  other  energy 
production  and  distribution  alternatives  (National  Commission  on  Energy  Policy,  2004; 

Energy  Information  Administration,  2005).  As  noted  previously,  it  also  provided  an 
interesting  case  because  of  the  unique  risks  and  challenges  associated  with  the  integration 
of  LNG  within  the  current  U.S.  energy  supply  infrastructure,  given  the  high  level  of  concern 
surrounding  homeland  safety  and  security  (Hightower,  et  al.,  2004;  Clarke,  2005;  Aspen 
Environmental  Group,  2005).  Accordingly,  it  was  estimated  that  the  LNG  Scenario  was  apt  to 
be  of  topical  interest  to  study  participants. 

Although  no  formal  measures  were  taken  to  quantify  how  well  the  scenario  met  the 
selection  criteria,  again  due  to  privacy  concerns,  informal  feedback  indicated  that 
participants  judged  the  scenario  to  be  moderately  representative  of  intelligence  analysis. 
Comments  on  topic  authenticity  were  mixed.  As  expected,  no  participant  indicated  that  the 
scenario  was  exactly  like  their  work  as  an  analyst.  Most  indicated  that  they  found  the 
scenario  to  be  representative  of  their  work,  though  a  few  commented  that  they  did  not  find 
the  scenario  to  be  highly  generalizable.  Most  participants  also  found  the  scenario  to  be 
acceptable  in  its  use  of  unclassified  information  and  in  its  scope.  However,  the  LNG  Scenario 
prompted  mixed  reactions  in  terms  of  topical  interest,  with  some  participants  expressing  a 
general  interest  in  the  topic,  while  others  expressed  indifference. 

3.2.4  Design 

Once  selected,  the  study  scenario  was  developed  and  refined  based  on  inputs  from 
information  analysis  researchers,  professional  intelligence  analysts,  and  a  LNG  industry 
domain  expert.  Because  it  was  scenario-based,  the  two  sets  of  products  and  processes  used 
in  the  study  were  not  taken  directly  from  any  specific  analysis.  Rather,  they  were 
representative  analytic  products  and  processes,  based  on  a  real  world  case,  designed  to 
amplify  the  cues  analysts  use  to  infer  rigor — with  one  designed  to  reflect  a  high-rigor 
analysis  process  and  the  other  a  low-rigor  process. 

The  analysis  briefing  reports  for  each  junior  analyst  in  the  scenario  were  constructed 
through  collaboration  (Appendix  A,  Figures  A.2-A.5).  The  briefing  report  for  Analyst  1  was  a 
unique  analytical  response  developed  in  collaboration  with  an  LNG  industry  domain  expert 
who  was  a  critical  contributor  to  the  Battery  Rock  LNG  risk  assessment  report  (Clarke,  2006). 
The  briefing  report  for  Analyst  2  was  heavily  based  on  an  unclassified  strategic  forecast  on 
the  future  of  LNG  in  the  U.S.  (STRATFOR,  2005).  Both  reports  were  critiqued  by  a  former 
intelligence  analyst  during  pre-testing  of  the  study  methods,  resulting  in  significant 
revisions  to  the  analysis  reports. 
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Difference  Description 


Excerpt  from  A(1)  Report  Excerpt  from  A(2)  Report 


Tone  of 
Reports 


The  two  briefings 
conveyed  distinctly 
different  tones  to  the 
reader.  Most  notably, 
the  reports  revealed  a 
difference  in  perspective 
between  author 
analysts.  While  Analyst  1 
did  not  support  the 
position  that  importing 
LNG  is  an  appropriate 
response  to  the 
increasing  U.S.  energy 
need.  Analyst  2  favored 
LNG  as  a  viable  solution. 
In  addition,  the  reports 
differed  in  tone  relative 
to  writing  style  as  well 
as  in  the  subtle 
differences  in  briefing 
format  and 
organization. 


"Terrorist  groups...  have  the 
intent  and  capability  to 
attack  urban  LNG  facilities... 
It  is  unlikely  that  deterrence 
or  prevention  measures 
would  be  adequate  to 
defend  against  attacks" 


"Rising  gasoline  prices,  oil 
price  volatility,  and  the 
possibility  of  domestic  oil 
shortages  have  increased  U.S. 
demand  for  natural  gas... 
Domestic  natural  gas 
reserves  are  no  longer 
sufficient  to  satisfy  the 
growing  demand" 


"There  are  four  primary 
obstacles..." 

"The  goals  publicly 
articulated  by  these  groups 
include..." 

"[M]easures  to  reduce  the 
risk  or  attacks  on  LNG 
facilities  include..." 


"[L]iquified  natural  gas... 
provides  a  way  to  both 
plug  the  gap  and  bring 
down  prices  from  their 
recent  highs...  the  politics, 
security  concerns  and 
economics  of  LNG  and  the 
United  States'  energy  needs 
match" 

"[N]ot  all  is  well  in  the 
world  of  natural  gas  in  the 
United  States...  In  times 
past,  U.S.  natural  gas 
demand  was  entirely 
satisfied  by  domestic,  and 
then  Canadian,  production. 
That  time  has  ended...  If  a 
long-term  solution  is  not 
adopted  quickly,  wide 
swathes  of  U.S.  industry  will 
simply  cease  functioning" 

"Luckily  for  the  United 
States..." 

"Unfortunately  for  the 
United  States..." 

"LNG  is  more  than  merely 
cost  competitive.  It  is  far 
cheaper  than  American... 
piped  natural  gas." 


Obstacle 

Set 


The  two  briefings 
differed  in  the  obstacles 
that  were  cited  in  each 
report  as  those  primary 
in  prohibiting  the  more 
widespread  acceptance 
of  LNG.  The  reports  had 
only  partially 
overlapping  obstacle 
sets. 


"1)  risk  of  fire  and  radiation 
release  from  intentional 
terrorist  attacks...  2)  risk  of 
fire  and  radiation  release 
from  unintentional  release... 
3)  resistance  to  the  use  of 
lowly  populated 
environmental  areas  for 
industrial  purposes,  and  4) 
reduced  property  values  to 
local  areas" 


"[T]he  United  States...  lacks 
a  unified  energy  policy 
capable  of  addressing  long¬ 
term  issues.  That  has 
allowed  local  —  as  opposed 
to  national  — 
environmental  groups 
effectively  to  stall  the 
development  of  LNG 
import  facilities" 


Inclusion  The  A(2)  report  included  No  figure, 
of  Figure  a  line  graph  related  to 
LNG,  while  A(1)  did  not. 

Because  the  figure  was 
included  in  the  low-rigor 
analysis  product,  note 
that  the  graph  was  not 
directly  relevant  to  the 
primary  task  question. 


Table  3.  Designed  Differences  Between  Analysis  Products  (Briefing  Reports) 
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Process  Document 

Description 

Analysis  Process  1 

Analysis  Process  2 

Query  Summary 

While  the  total  number  of  queries 
did  not  differ  substantially  between 
the  two  processes,  the  type  of 
queries  used  in  each  process  did.  A 
(1)  used  more  targeting  queries  and 
broader  queries,  while  A(2)  tended 
to  use  search  queries  that  were 
focused  on  a  main  hypothesis. 

4  query  targets 

3  query  targets 

Documents  Read 
(in  order  of 
reading) 

A(1)  and  A(2)  had  a  nearly  four-fold 
difference  in  number  of  documents 
read.  The  number  of  documents 
read  reflected  both  on  the  overall 
rigor  of  the  analytic  process,  but 
also  on  the  experience  and 
knowledgeability  of  the  author 
analysts. 

26  documents 

7  documents 

Where  Documents 
Stored  in  Folders 

The  two  processes  differed  on  the 
extent  to  which  documents  were 
organized  during  the  analysis 
process  and  also  in  the  approach 
used  to  organize  the  document  sets. 
The  A(1)  documents  were  better 
organized. 

7  folders 

4  folders 

Key  Documents 
(relied  on  heavily) 

The  nature  of  document  identified 
as  key  differed  between  processes. 

In  A(1),  the  key  documents  were 
classified  as  formal  research  reports 
whereas  in  A(2)  tended  to  rely  on  a 
varied  mix  of  less  formal  supporting 
documents. 

4  key  documents 

3  key  documents 

Hypotheses  — 
Considered 
Obstacles  to  LNG 

A(1)  reflected  one  in  which  more 
hypotheses  were  weighed  and 
considered.  Additionally  the  nature 
of  the  hypotheses  that  were 
investigated  varied  between  the 
two  processes. 

5  hypotheses 

3  hypotheses 

Collaborations 
with  Others  During 
Process 

A(1)  showed  a  substantially  greater 
the  incorporation  of  alternative 
perspectives  through  collaboration. 

It  included  both  more  and  better 
quality  collaborations  than  A(2). 

6  collaborations 

1  collaboration 

Research  Note- 
Sheet 

For  A(1),  study  participants  were 
able  to  view  the  collection  of  notes 
the  analysts  used  in  preparing  the 
final  briefing  report.  In  contrast,  the 
A(2)  analyst  did  not  use  a  process 
that  included  a  note-sheet. 

Note-sheet 

Not  used 

Note.  Each  of  the  Process  Documents  listed  in  the  first  column  of  this  table  is  included  in  an  Appendix  A. 


Table  4.  Designed  Differences  Between  Analysis  Processes  (Process  Documents) 
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In  order  to  ensure  that  participants  would  have  enough  time  to  critically  evaluate  both  of 
the  reports,  each  briefing  was  limited  two  pages  in  length.  Report  1,  designed  to  be  a  high- 
rigor  analysis  product,  was  created  by  combining  perspectives  from  a  number  of  different 
LNG  reports.  Report  2,  in  contrast,  was  created  by  abridging  the  single  perspective  in  the 
strategic  forecast  report  to  reflect  a  low-rigor  analysis  product.  The  briefing  reports  were 
then  further  refined  to  differ  with  respect  to  the  facets  of  tone  of  reports,  obstacle  set,  and 
inclusion  of  figure,  as  shown  in  Table  3. 

Concurrent  with  the  development  of  the  briefing  reports,  the  analysis  processes  that 
produced  the  reports  were  meticulously  constructed  (Appendix  A,  Figures  A.7-A.40).  As  was 
the  case  for  the  briefing  reports,  the  analysis  process  of  Analyst  1  was  designed  to  reflect  a 
high  rigor  process,  while  the  analysis  process  of  Analyst  2  was  designated  as  a  low  rigor 
process.  In  the  case  of  A(1),  this  was  done  by  refining  aspects  of  the  actual  process  used  to 
create  the  report.  In  A(2),  the  process  was  constructed  to  describe  the  briefing  product  that 
was  created  in  paring  down  the  strategic  forecast,  with  respect  to  the  stated  goal  of 
representing  it  as  a  low-rigor  process. 

In  the  context  of  the  study,  analytic  processes  were  reflected  by  a  set  of  process  documents. 
Process  documents  are  a  collection  of  descriptive  documents  that  provide  insight  into  an 
analysis  process,  with  each  document  in  the  set  revealing  partial  insight.  As  an  aggregate, 
this  document  set  affords  the  construction  of  a  model  of  the  analytic  processes  that 
produced  an  analysis  report.  Process  documents  were  used  in  the  study,  rather  than  other 
methods  of  providing  process  insight,  because  they  offered  a  compromise  between 
providing  raw  data  about  process  (e.g.,  the  recording  and  replaying  of  an  analysis  process) 
and  providing  interpretations  of  process  (e.g.,  a  descriptive  summary  of  a  process),  while  not 
requiring  participants  to  have  a  direct  interaction  with  the  producing  analyst  (e.g.,  a 
discussion  with  the  author  analyst  about  their  analytic  process). 

Although  there  are  many  potential  types  of  process  documents  that  could  be  used  to 
describe  an  analysis  process,  seven  specific  process  documents  were  selected  for  inclusion  in 
the  study.  Two  sets  of  process  documents,  listed  in  Table  4,  were  developed  to  be  used  by 
participants  in  forming  an  understanding  of  the  analysis  processes.  The  seven  specific 
document  types  were  selected  because  they  were  deemed  effective  in  capturing  the  ways  in 
which  the  analysis  processes  in  the  scenario  differed  with  respect  to  assessment  of  rigor. 
Additionally,  some  of  the  document  types  were  selected  for  inclusion  based  on  their 
potential  to  be  collected  electronically  and  automatically  during  the  course  of  an  analysis. 


3.3  Supervisor's  Dilemma 

This  research  was  driven  in  part  by  a  critical  judgment  task  referred  to  as  the  Supervisor's 
Dilemma.  The  Supervisor's  Dilemma  describes  a  generic  situation  wherein  a  supervisor  must 
decide  if  the  output  product  of  an  analyst  is  acceptably  rigorous  or  if  more  resources  must 
be  invested  in  the  analysis  process  before  sending  it  forward.  This  judgment  provides  an 
interesting  mechanism  for  exploring  the  understanding  of  rigor  in  analysis  because,  while  in 
principle  it  is  an  abstraction  of  a  common  occurrence,  it  represents  a  critical  decision  that,  in 
practice,  is  often  made  tacitly  (Betts,  1978).  Moreover,  it  captures  the  spirit  of  many  related 
questions  that  a  supervisor  might  ask  about  an  analysis  product — Is  the  analysis  ready  to  send 
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forward?  If  not,  what  else  needs  to  be  known  ?  Given  what  Is  not  known,  what  is  the  best  way  to 
invest  limited  analytical  resources  to  reduce  this  uncertainty  and  improve  the  analysis  product? 

The  Supervisor's  Dilemma,  however,  represents  more  than  just  a  series  of  binary  questions  a 
supervisor  could  ask  about  an  analysis.  Rather,  the  dilemma  highlights  the  criticality  of  the 
interactions  that  occur  at  the  interface  between  the  analysis  and  replanning  components  of 
information  analysis.  It  represents  a  point  in  the  intelligence  cycle  where  a  decision  is  made 
— whether  by  supervisors,  decision  makers,  or  the  analysts  themselves — as  to  whether  or  not 
an  analysis  product  is  ready  to  pass  on  to  the  next  stage  in  the  cycle.  Simply,  whether  made 
implicitly  of  explicitly,  the  Supervisor's  Dilemma  addresses  a  basic  question  in  information 
analysis — When  is  analysis  sufficient?  Thus,  the  Supervisor's  Dilemma  also  serves  to  increase  the 
generality  of  the  study  design  beyond  intelligence  analysis  alone,  as  the  ubiquity  of  judging 
sufficient  rigor  is  a  task  fundamental  to  all  forms  of  information  analysis. 

Framing  the  Supervisor's  Dilemma  as  a  generic  judgment  task  also  circumvents  some  of  the 
controversies  in  the  intelligence  community  surrounding  the  role  of  the  decision  maker 
relative  to  the  analysis  process.  Structuring  this  critical  judgment  as  a  dilemma  of  the 
supervisor  frames  the  decision  as  one  that  is  explicit,  collaborative,  and  yet  largely  informal. 
The  dilemma  is  not  so  personal  that  the  decision  is  made  implicitly  by  an  analyst  and  yet  not 
so  formal  that  the  analysis  is  used  by  a  decision  maker  as,  perhaps,  the  sole  basis  for  a  policy 
decision.  Thus,  the  role  of  supervisor  reflects  a  perspective  somewhere  between  that  of 
analyst  and  that  of  decision  maker,  providing  an  agreeable  point  on  the  continuum  for 
eliciting  feedback  from  study  participants  about  how  they  judge  rigor  in  the  analysis 
process. 

The  Supervisor's  Dilemma  spanned  across  the  phases  of  the  study,  as  participants  were  asked 
to  make  an  assessment  of  analysis  report  readiness  both  before  and  after  gaining  insight 
into  the  processes  that  produced  the  reports.  In  the  study,  the  dilemma  was  framed,  "As  a 
senior  analyst,  you  might  be  asked  to  decide  if  junior  analysts'  reports  are  ready  to  be  passed  on  to  a 
policy-maker"  (Appendix  A,  Figure  A.1).  The  dilemma  facing  the  study  participants,  as  proxy 
supervisors,  was  to  decide  if  either,  both,  or  neither  of  the  scenario-based  analysis  briefings 
were  ready  to  be  forwarded  on  to  the  decision  maker,  by  assessing  the  rigor  of  the  processes 
that  produced  them.  Thus,  the  Supervisor's  Dilemma  put  participants  into  an  information 
analysis  role  that  highlights  the  criticality  of  the  concept  rigor. 


3.4  Procedure 

The  design  of  this  study  was  based  on  the  Elicitation  by  Critique  (EBC)  methodology  (Miller, 
Patterson,  &  Woods,  2006).  In  the  EBC  approach,  participants  share  their  expertise  by 
critiquing  the  processes  of  other  domain  practitioners,  rather  than  by  directly  verbalizing 
their  individual  experiences.  EBC  is  also  similar  to  the  individual  allo-confrontation  method 
in  that  both  approaches  confront  participants  "with  an  activity  they  practice  but  which  is 
performed  by  someone  else"  and  prompt  them  to  verbalize  about  that  activity  (Mollo  & 
Falzon,  2004). 

The  findings  of  this  study  support  the  EBC  method  as  a  valuable  approach  for  taping  into 
the  expertise  of  specialist  practitioners,  while  simultaneously  refining  the  method,  as  the 
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approach  used  in  this  study  was  modified  from  the  original  EBC  methodology  in  three  ways. 
First,  participants  critiqued  not  only  the  processes  of  other  analysts,  but  also  the  briefing 
products  that  were  produced  by  those  processes.  Second,  participants  critiqued  the  analyses 
of  two  different  analysts  concurrently,  rather  than  sequentially.  Third,  and  perhaps  most 
significantly,  participants  critiqued  analyses  that  were  generated  to  fit  within  the  context  of 
the  LNG  Scenario.  Due  to  these  changes  from  the  original  method,  the  design  of  this  study 
approach  was  a  modified  EBC  methodology,  embedded  in  a  scenario  walkthrough. 

Each  participant  engaged  in  a  two-part  critique  during  a  single  elicitation  session.  In  the 
first  part  of  the  study,  critiquing  focused  on  the  analysis  products,  while  in  the  second  it 
focused  on  the  analysis  processes.  The  sessions  also  included  an  introductory  overview  of  the 
study  and  a  concluding  debrief  discussion. 

A  summary  of  the  session  protocol  is  found  in  Table  5.  Note  that  the  elicitation  session 
procedure  served  as  a  tentative  outline  with  respect  to  the  amount  of  time  spent  on  each 
step.  Every  participant  progressed  through  all  steps;  however,  in  many  cases  there  were 
deviations  from  the  baseline  time  estimates,  as  dictated  by  the  nature  of  the  discussion 
generated  by  individual  participants. 


Phase 

Step 

Estimated  Time 
(minutes) 

Introduction 

Consent  to  Participate  Obtained 

- 

Introduction  to  Study 

5 

Product  Critique 

Introduction  to  Scenario 

5 

Participants  Review  Briefing  Reports 

15 

Discussion  of  Differences  Between  Products 

20 

Supervisor's  Dilemma  Assessment 

- 

Process  Critique 

Discussion  of  Value  of  Process  Documents  in  Inferring  Rigor 

10 

Participants  Review  Process  Documents 

15 

Discussion  of  Differences  Between  Processes 

15 

Supervisor's  Dilemma  Reassessment 

- 

Conclusion 

Study  Debrief 

5 

Years  of  Experience  and  Scenario  Familiarity  Obtained 

- 

Table  5.  Elicitation  Session  Process 

Each  of  the  elicitation  sessions  was  conducted  with  participants  individually,  with  either  one 
or  two  researchers  present.  The  first  five  sessions  included  two  researchers  while  the  final 
four  sessions  each  had  one.  The  same  two  researchers  were  involved  in  all  sessions,  with 
participant  feedback  primarily  collected  as  verbal  report  data  in  the  form  of  written  notes. 
With  the  exception  of  the  pilot  participant,  audio  recordings  of  all  elicitation  sessions  were 
collected  for  later  analysis  using  an  Apple  iPod  equipped  with  a  Griffin  iTalk  Voice  Recorder. 
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The  elicitation  sessions  were  conducted  in  unclassified  conference  rooms,  on-site  at  the 
home  campus  of  the  participants'  respective  intelligence  organizations.  Each  session  was 
scheduled  for  one  and  a  half  hours,  with  the  exception  of  P8,  whose  session  was  limited  to 
one  hour  due  to  time  constraints.  The  study  sessions  were  run  consecutively  over  a  four  day 
period  at  times  convenient  for  the  participants. 

The  introductory  overview  focused  on  providing  participants  with  a  general  understanding 
of  the  goals  of  the  research  study  as  well  as  a  description  of  what  participation  in  the 
elicitation  session  would  entail.  The  primary  objective  of  the  study,  as  described  to 
participants,  was  to  discover  the  cues  that  expert  intelligence  analysts  used  to  assess  the 
level  of  rigor  of  an  analytic  process.  Further,  it  was  explained  that  participant  inputs  and 
assessments  would  be  used  to  gain  insight  into  the  primary  objective  of  discovery  of  rigor 
cues. 

In  accomplishing  these  objectives,  participant  analysts  were  asked  to  engage  in  an 
elicitation  session  in  which  they  would  analyze  a  scenario  by  (1)  reading  two  written  reports 
and  comparing  their  quality,  (2)  assessing  and  comparing  the  rigor  of  the  processes  used  to 
create  each  of  the  reports,  and  (3)  participating  in  an  open-ended,  unstructured  discussion 
of  the  cues  used  to  assess  analytic  rigor.  General  questions  about  the  study  were  addressed 
at  this  time  and  verbal  consent  to  participate  was  obtained  from  all  participants,  as  well  as 
consent  to  record  the  discussion.  All  potential  participants  agreed  to  participate  in  the  study 
and  to  be  recorded. 

During  the  initial  critique  cycle — focused  on  the  analysis  products — participants  were  first 
given  a  verbal  description  of  the  LNG  Scenario.  The  authenticity  of  the  scenario  was  also 
briefly  commented  upon,  and  the  selection  criteria  used  in  choosing  the  scenario  were 
explained.  Participants  were  then  given  a  one-page  summary  that  introduced  the  study 
scenario,  found  in  Appendix  A,  that  succinctly  defined  their  role  as  hypothetical  supervisor 
in  the  context  of  the  scenario. 

Participants  were  also  asked  to  remember  the  following  during  the  scenario  study:  (1)  There 
is  no  "right"  answer  for  the  scenario.  It  is  not  a  test  of  knowledge  of  LNG  or  the  current  U.S. 
energy  situation.  Rather,  the  scenario  is  intended  to  help  reveal  the  indicators  used  to  assess 
rigor.  (2)  Avoid  focusing  too  much  on  how  this  scenario  differs  from  what  they  do  as 
analysts.  Instead,  it  should  be  used  as  a  facilitator  for  exploring  the  dimensions  of  rigor.  (3) 
Personal  insights  provide  the  grounding  needed  to  explore  design  concepts  that  could 
meaningfully  impact  the  way  analysis  is  done  in  the  future.  Thus,  the  more  discussion 
provided  by  participants,  the  more  valuable  was  the  elicitation  session. 

Participants  were  next  presented  with  the  two  analysis  briefings  generated  via  Analysis 
Process  1  [A(1 )]  (Appendix  A,  Figures  A.7-A.33)  and  Analysis  Process  2  A(2)  [A(2)j  (Appendix 
A,  Figures  A.34-A.40).  Note  that  the  order  in  which  the  briefings  were  presented  was 
randomized  across  sessions,  such  that  for  some  participants  A(1)  was  labeled  as  A(2),  and 
vice  versa.  However,  for  the  sake  of  clarity,  throughout  this  paper  A(1)  always  refers  to  the 
high-rigor  analysis  process  and  A(2)  always  refers  to  the  low-rigor  analysis  process. 
Participants  were  instructed  to  read  both  reports,  keeping  in  mind  that  they  would  be  asked 
to  "assess  and  compare  the  quality  of  the  two  written  reports"  and  "comment  on  the  'rigor'  of  the 
processes  that  produced  each  of  the  reports"  (Appendix  A,  Figure  A.1). 
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Participants  were  then  given  up  to  fifteen  minutes  to  individually  review  the  two 
documents,  though  some  took  less  than  than  the  full  amount  of  time  alloted.  Each 
participant  was  provided  with  means  to  take  notes  about  the  reports  as  they  deemed 
appropriate.  They  were  also  informed  that  they  could  mark  directly  on  each  briefing  report. 
During  this  time,  participants  were  left  alone  while  they  reviewed  the  analysis  reports.  After 
participants  completed  their  review  of  the  analysis  briefings  and  were  prepared  to  discuss 
them,  they  signaled  for  the  researchers  return. 

At  this  time,  participants  were  prompted  to  discuss  the  differences  between  the  analysis 
products  and  critique  the  two  briefings  by  (1)  comparing  and  contrasting  them,  (2) 
commenting  on  what  they  could  infer  about  the  rigor  of  the  process  that  produced  each 
report,  and  (3)  describing  the  approach  they  would  use  to  better  understand  the  rigor 
behind  each  report.  In  addition  to  pursuing  these  structured  lines  of  inquiry,  the  researchers 
also  interacted  with  participants  to  clarify  comments  and  request  elaboration  on  topics  of 
interest  raised  in  the  course  of  the  discussion.  The  first  critique  cycle  concluded  when 
participants  were  asked,  in  their  role  within  the  scenario  as  analyst  supervisor,  to  decide  if 
either,  both,  or  neither  of  the  analysis  briefings  were  ready  to  be  forwarded  on  to  the 
decision  maker — thus  putting  themselves  into  the  Supervisor's  Dilemma. 

The  second  critique  cycle — focused  on  assessing  the  analysis  process — began  when 
participants  were  provided  with  a  list  of  process  documents  (Appendix  A,  Figure  A. 6)  that 
could  potentially  be  used  in  assessing  analytic  rigor.  They  were  then  asked  to  comment  on 
the  perceived  value  of  having  access  to  each  type  of  process  document  in  making  an 
assessment  of  analytic  rigor,  ranking  the  seven  options.  In  addition  to  verbal  report  data, 
ordinal  preference  rankings  were  collected  from  participants. 

Next,  participants  were  given  access  to  two  sets  of  process  documents,  reflecting  A(1)  and  A 
(2)  (Appendix  A,  Figures  A.7-A.40).  As  in  the  first  critique  cycle,  participants  were  then  given 
time  alone,  up  to  fifteen  minutes,  to  review  the  process  documents.  Before  reviewing  the 
documents,  participants  were  advised  that  they  were  not  required  to  read  the  entirety  of 
information  that  was  made  available,  but  rather  that  they  could  review  the  information 
that  seemed  most  critical  to  building  an  understanding  of  the  analysis  processes  that 
generated  each  briefing  report. 

After  examining  the  process  documents  and  signaling  for  the  researchers  to  return, 
participants  were  asked  to  critique  the  analysis  processes  by  (1)  comparing  and  contrasting 
the  processes,  (2)  commenting  on  the  rigor  they  saw  in  each  process,  and  (3)  comparing 
their  revised  assessments  of  rigor — based  on  insight  into  the  actual  analysis  processes — to 
their  prior,  perceived  assessments  of  rigor — made  based  on  inferences  from  the  analysis 
products  alone.  Once  again,  the  researchers  interacted  with  participants  to  explore  related 
topics  beyond  these  initial  discussion  points.  The  second  critique  cycle  concluded  with  a 
reassessment  of  the  Supervisor's  Dilemma,  with  participants  now  able  to  take  into 
consideration  a  more  complete  understanding  of  the  rigor  of  the  analysis  processes. 

During  the  concluding  debrief  discussion,  data  was  collected  from  participants  regarding 
two  specific  questions.  First,  participants  were  asked  to  provide  an  estimate  of  their  years  of 
relevant  professional  experience.  Second,  they  were  asked  to  disclose  if  they  had  any 
previous  professional  or  personal  experience  related  to  the  importation  of  LNG  that  might 
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bias  their  perspective  toward  the  scenario.  No  participant  indicated  any  significant 
background  related  to  the  scenario  topic. 

Participants  were  also  asked  to  identify  what  they  perceived  to  be  the  most  significant 
challenge  currently  impacting  the  intelligence  analysis  community,  prompting  a  wide  variety 
of  responses.  The  debrief  also  provided  an  opportunity  to  comment  on  the  design  of  the 
study,  which,  as  noted  previously,  some  participants  did.  Finally,  it  provided  participants  with 
a  chance  to  ask  questions  related  to  their  participation  in  the  research.  Other  outstanding 
concerns  regarding  the  study  were  also  addressed  at  this  time. 

Following  the  completion  of  the  study,  the  data  generated  in  the  form  of  written  notes 
were  e-mailed  to  participants,  via  the  sponsor  contact,  for  review.  Copies  of  the  notes  sent 
to  participants  are  found  in  Appendix  B,  Figures  B.1-B.29.  This  provided  participants  with  an 
opportunity  to  verify  that  they  did  not  divulge  any  classified  information  during  their 
elicitation  session,  to  correct  any  inaccuracies,  and  to  offer  any  additional  insights  that  may 
have  emerged  after  the  completion  of  the  study.  To  date,  no  revisions  or  corrections  to  the 
original  study  data  have  been  requested. 


3.5  Data  Analysis  Methodology 

The  primary  mechanism  for  eliciting  comparable  feedback  across  study  participants  was  the 
prompt.  The  data  collected  during  the  sessions  was  analyzed  by  organizing  the  participant 
feedback  relative  to  four  such  prompts  designed  into  the  study,  which  included  two 
discussion  prompts  and  two  decision  prompts. 

The  two  discussion  prompts,  comparison  of  briefing  reports  and  comparison  of  process 
documents,  were  part  of  the  study  critique  cycles — in  the  product  critique  cycle  and  the 
processes  critique  cycle,  respectively.  These  prompts  served  to  elicit  responses  from 
participants  about  the  cues  they  used  to  infer  rigor  relative  to  the  various  documents  they 
interacted  with  throughout  the  study.  The  two  decision  prompts,  rigor  assessment  and 
Supervisor's  Dilemma,  spanned  across  multiple  phases  of  the  study  and  were  designed  to 
capture  how  process  insight  influenced  perceptions  of  rigor — relative  to  analysis  process  and 
product,  respectively. 

The  comparison  of  briefing  reports,  the  first  discussion  prompt  that  participants 
encountered  in  the  study,  focused  on  how  participants  responded  to  the  differences 
between  the  briefings.  At  the  outset  of  the  study,  participants  understood  that  the  two 
versions  of  the  analysis  briefing  were  created  by  two  similar  analysts  working  the  same 
questions  under  similar  constraints.  Yet,  the  reports  differed  significantly  in  their  responses 
to  the  LNG  Scenario  task  question,  requiring  participants  to  resolve  this  discrepant 
information. 

The  differences  between  the  two  reports  was  reflected  in  the  interaction  of  the  three  facets 
of  tone  of  reports,  obstacle  set,  and  inclusion  of  figure,  as  described  in  the  previous  section 
(Table  3).  The  facets  also  represented  the  three  embedded  probes  used  to  organize  the 
study  data  relative  to  comparisons  of  the  briefing  reports.  Embedded  probes  are  latent 
aspects  of  the  scenario  that  serve  to  push  it  beyond  a  standard  case  (Woods  &  Flollnagel, 
2006).  Probes  differ  from  prompts,  however,  in  that  participants  were  not  explicitly  asked  to 
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comment  on  or  respond  to  them.  Rather,  embedded  probes  become  informative  by 
understanding  which  probes  participants  actually  did  attend  to  during  the  course  of  the 
study.  Each  of  two  discussion  prompts  built  upon  a  set  of  such  probes. 

In  comparing  the  process  documents  that  described  each  briefing  report,  participants  used 
the  available  documents  in  building  an  understanding  of  process.  In  the  study,  the  two 
analysis  processes  were  designed  to  be  distinctly  different,  which  was  reflected  in  the 
process  documents.  One  analysis  process  was  performed  via  what  was  characterized  as  a 
high-rigor  process,  while  the  other  was  performed  via  a  low-rigor  process.  This  distinct 
difference  between  analyses  again  required  study  participants  to  cope  with  discrepant 
information.  The  differences  between  analyses  were  manifested  in  the  process  documents 
available  to  participants.  Each  of  these  seven  unique  types  of  process  documents,  listed  in 
Table  4,  also  served  as  an  embedded  probe  in  the  study.  As  in  the  comparison  of  reports 
prompt,  the  use  of  embedded  probes  further  structured  how  study  data  were  organized 
and  subsequently  analyzed. 

Decision  prompts  were  distinct  from  the  discussion  prompts  in  that,  rather  than  reflecting 
specific  phases,  they  were  recurrent  across  phases  of  the  study.  The  decision  prompts  were 
instantiated  when  participants  made  decisions  both  before  and  after  gaining  insight  into 
the  analysis  processes.  Additionally,  unlike  discussion  prompts,  decision  prompts  asked 
participants  to  make  explicit  judgments — by  indicating  which  report  or  process  was 
perceived  as  being  better,  rather  than  qualitatively  comparing  and  contrasting  them. 

The  rigor  assessment  prompt  asked  participants  to  decide  which  analysis  process  they 
believe  to  be  more  rigorous,  both  before  and  after  reviewing  the  process  documents.  In 
assessing  the  analysis  process  before  seeing  the  process  documents,  participants  were 
limited  to  what  could  be  inferred  from  the  briefing  reports  in  making  assessments  of 
analytic  rigor.  The  prompt  was  fully  realized  when  participants  viewed  the  actual  process 
documents,  which  allowed  them  to  reinterpret  prior  assessments  about  report  quality 
relative  to  the  new  information  that  had  been  revealed.  This  prompt  required  participants 
to  quickly  shift  from  thinking  abstractly  about  analytic  rigor,  based  on  inference,  to  thinking 
very  concretely  about  rigor,  based  on  explicit  evidence  found  in  the  process  documents. 

The  Supervisor's  Dilemma  also  served  as  a  prompt  in  that  it  challenged  participants  to  assume 
a  role  that  they  may  not  have  on  a  regular  basis — that  of  a  supervisor  who  must  decide  if  an 
analysis  product  is  ready  to  go  forward  to  a  decision  maker.  This  prompt  spanned  across  the 
phases  of  the  study,  as  participants  were  asked  to  make  assessments  of  analysis  report 
readiness  after  reviewing  the  briefing  reports  and  after  reviewing  the  process  documents.  In 
doing  so,  participants  were  confronted  with  a  situation  wherein  the  new  information  has 
either  supported  or  contradicted  their  initial  judgments  of  the  rigor  of  the  analysis  products. 
Their  attempt  to  resolve  this  potential  second-level  dilemma  provided  insight  into  their 
understanding  of  how  process  rigor  relates  to  product  quality. 
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3.6 


Study  Findings 


A  number  of  interesting  findings  emerged  from  the  analysis  of  the  data  collected  in  the 
study.  These  results  are  described  in  this  section,  with  the  discussion  of  the  broader 
implications  of  these  findings  is  covered  in  the  next  section.  The  findings  are  organized  by 
prompts,  as  described  in  the  data  analysis  methodology.  Also,  throughout  this  section  direct 
quotes  from  the  analysts  who  participated  in  the  study  appear  centered  and  in  italics, 
revealing  additional  insight  into  the  perspective  of  the  professional  information  analyst. 

3. 6. 1  Comparison  of  Briefing  Reports  Prompt 

The  comparison  of  briefing  reports  prompt  organized  participant  data  around  responses 
that  compared  and  contrasted  the  scenario  briefing  reports.  Organizing  the  data  relative  to 
this  prompt  revealed  that  participants  generally  made  qualitatively  similar  assessments  of 
the  A(1)  and  A(2)  briefings.  This  conclusion  was  supported  across  report  comparison  probes, 
as  participants  expressed  similar  conclusions  about  both  analysts  and  analysis  processes 
based  solely  on  the  briefing  reports.  These  findings  are  organized  with  respect  to  the  three 
embedded  probes,  each  of  which  structured  the  participant  data  around  a  different  facets 
of  report  comparison — including  tone  of  reports,  obstacle  set,  and  inclusion  of  figure. 

The  first  probe  explored  in  the  comparison  of  briefing  reports  was  based  on  the  comments 
made  regarding  the  tone  of  the  briefing  reports.  This  probe  was  broken  down  into  three 
components,  each  of  which  aggregate  certain  types  of  feedback  from  participants  about 
tone.  The  three  components  include  analyst  stance,  writing  style,  and  briefing  form. 


ID 

A(1)  Briefing  Report 

A(2)  Briefing  Report 

PO 

Against  LNG 

Bias  toward  terrorism  issues 

Supports  LNG 

Views  economy  as  central  problem 

PI 

Good  hypotheses,  but  too  narrow 

Energy  and  terrorism  mindset 

Poor  hypotheses,  but  more  broad 

P2 

Apparent  anti-LNG  bias 

Terrorism  is  bigger  than  it  should  be 

Apparent  pro-LNG  bias 

Written  to  support  an  opinion 

P3 

Focus  is  terrorism,  but  did  address 
other  issues 

Focus  is  environmental  groups 

P4 

Terrorism  focus  is  off  topic 

Does  not  show  a  strong  bias 

- 

P5 

- 

Has  a  political  agenda 

Assumes  reader  holds  same  views 

P6 

Threat  of  terrorism  focus 

Economic  focused 

P7 

Has  a  bias  toward  terrorism 

Seems  to  be  a  data  gatherer 

P8 

Has  a  social  orientation 

More  math  /  economic  oriented 

Table  6.  Summary  of  Responses  to  Comparison  of  Reports  Prompt,  Tone  of  Reports  Probe, 
commenting  on  perceptions  of  author  Analyst  Stance 
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'It's  not  analysis  to  justify  your  biases. 


The  general  perspective  among  participants  was  that,  when  assessing  analysis  products,  it  is 
important  to  consider  the  possibility  that  the  stance  of  the  author  analyst  can  bias  an 
analysis  process.  Regarding  the  scenario,  the  general  opinion  of  participants  was  that 
neither  report  appeared  entirely  unbiased.  Table  6  shows  a  similarity  among  participants  in 
their  inferences  about  author  stance,  relative  to  the  broader  tone  of  reports  probe. 

Note  that  seven  of  nine  participants  commented  on  a  perceived  bias  in  Analyst  1  toward 
terrorism  as  the  primary  obstacle,  while  three  observed  a  bias  in  Analyst  2  toward  economic 
factors  as  the  central  obstacle.  PO  and  P2  went  even  further,  inferring  the  stances  of  the 
authors  on  the  task  question  and  identifying  Analyst  2  as  pro-LNG  and  Analyst  1  as  anti- 
LNG,  based  on  the  briefings  alone.  P2  went  further  still,  inferring  that  the  A(2)  briefing  was 
written  specifically  to  support  the  predisposition  of  the  decision  maker. 

"I  think  the  use  of  language  is  very  important. " 

Table  7  captures  those  responses  from  participants  about  the  tone  of  the  analysis  reports 
that  addressed  writing  style.  Aspects  of  writing  style  were  mentioned  by  all  participants 
during  their  comparisons  of  analysis  products.  The  nature  of  the  comments  indicated  a 
general  view  among  participant  analysts  that  the  use  of  language  was  a  critical  cue  for 
assessing  the  rigor  of  an  analytic  product. 

This  aspect  of  the  tone  of  reports  probe,  however,  yielded  a  more  variable  response  than 
aspects  of  analyst  stance.  There  was  a  split  among  participants  as  to  which  report  was 
perceived  as  having  the  better  writing  style,  with  P5  and  P7  indicating  that  the  A(1)  briefing 
was  better,  while  PO  and  P8  indicated  that  A(2)  yielded  a  better  briefing.  Similarly,  there  was 
disagreement  as  to  which  report  appeared  more  fact-based,  as  PI,  P2,  and  P5  found  the  A(1) 
report  to  be  more  factual,  while  PO,  P3,  and  P8  concluded  the  opposite,  identifying  the  A(2) 
report  as  more  sound. 

"Essentially,  if  you're  trying  to  answer  a  question  for  a  policy  maker,  they  want  the 
question  answered.  They  don't  really  want  you  to  present  them  with  the  facts  [that] 
cause  them  to  have  to  do  the  analysis.  In  other  words,  they  want  you  to  tell  them 
succinctly  what  the  answer  to  the  question  is — not  just  give  them  a  bunch  of  facts 
relating  to  the  question.  It's  a  matter  of  knowing  who  your  audience  is. " 

In  responding  to  the  organization  and  format  aspects  of  the  tone  of  reports  probe,  there 
was  a  clear  similarity  among  participants  regarding  what  was  viewed  as  most  important.  As 
shown  in  Table  8,  the  common  theme,  stressed  by  nearly  every  participant,  was  that  the 
format  of  the  analysis  briefing  should  focus  on  clearly  conveying  a  concise  answer  to  the 
decision  maker — not  just  provide  them  with  information  about  the  task  question.  A 
common,  related  sentiment — identified  explicitly  by  PO,  P2,  and  P7 — indicated  that  the  form 
of  the  analysis  briefing  should  be  tailored  specifically  to  the  preferences  of  the  analysis 
customer. 
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ID  A(1)  Briefing  Report  A(2)  Briefing  Report 


Got  to  the  point  much  quicker 

Good  instances  of  statistics  and  research 


PO  Too  much  background  information 

Never  left  the  first  level 

PI  Liked  fact-based  nature 

Likes  quantifying  of  what  was  and 
was  not  part  of  analysis 

Intermixes  solutions 


P2  Looks  like  parts  of  different  papers 
Looks  like  cutting  and  pasting 

P3  Did  not  like  use  of  terms  like  "regasified" 

It  took  away  from  credibility 

P4  Reads  like  a  newspaper  story  without 
supporting  documents,  sensationalistic 

Has  parts  that  do  not  help  policy-maker 

Not  technical  enough  to  show  analyst  knows 
his  stuff,  not  enough  detail 

P5  More  cohesive  explanations,  good  specificity 
Has  an  academic  style,  calm,  not  alarmist 


P6  Got  too  much  into  risk  mitigation 

P7  Covered  issues  more  thoroughly 

Tried  to  tell  a  story 
Pointed  out  areas  not  addressed 

P8  Very  scattered,  more  emotional 


Informal  writing  style  is  easier  to  read,  but 
something  is  lost  with  this  style 

Too  heavy  on  conjecture,  seems  opinionated 

Uses  words  like  "about"  and  "around" 
too  often 

Goes  off  task  by  offering  solutions, 
proposes  policy 


More  casual,  not  as  much  detail 
Tried  to  be  more  qualitative 

First  paragraph  is  not  on  subject 

Makes  unsupported,  extreme  statements 

Environmental  comments  seem  like  rhetoric 

Casual  style 

Confusing,  and  inappropriate  writing  style 
Few  or  no  facts,  based  on  opinion 

Like  an  "opinion  piece"  from  newspaper, 
alarmist 

Included  irrelevant  information 
Rambled  on  at  first,  went  off  topic 

All  over  the  map 

Picked  out  snippets,  cut  and  paste 

Not  as  well  as  written,  summarized  sources 

Tighter  writing,  more  factual 
Good  statistics  on  cost/benefit  analysis 


Table  7.  Summary  of  Responses  to  Comparison  of  Reports  Prompt,  Tone  of  Reports  Probe, 

commenting  on  Writing  Style 
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A(1)  Briefing 

A(2)  Briefing 

ID 

General  Comments  on  Organization  and  Form 

Report 

Report 

P0 

Tailoring  form  to  audience  is  key 

Form  should  be  "a  picture  and  a  paragraph" 

Policy  makers  are  looking  for  a  critical  idea  that  resonates 

Organization:  state  problem,  offer  a  solution,  and  state 
what  can  be  done  about  it,  but  should  not  "push  policy" 

PI 

Form  is  not  as  critical  unless  mandated 

Level  of  detail  reflects  the  type  of  report 

Should  be  one  page  for  policy  makers 

P2 

Form  is  key,  different  versions  of  a  report  are  useful 

Must  get  in  to  mind  of  customer,  customer  has  final  say 
Answer  must  be  clear,  easy  to  find,  focused  on  the  issue 
Should  include  what  is  not  known  and  what  to  expect 

Answered 

wrong 

question 

P3 

- 

- 

- 

P4 

First  paragraph  should  have  answer,  then  expand 

- 

Did  not 

Decision  makers  want  answers 

answer 

question 

P5 

Form  is  important 

- 

- 

Reader  must  have  confidence,  proper  form  conveys 
confidence 

Should  not  make  assumptions  about  the  reader 

Must  clearly  state  answer 

Should  attempt  to  qualify  what  is  not  taken  into 
consideration,  identify  the  limitations 

P6 

Reader  wants  to  know  what  their  position  should  be 

Got  off 

Did  not 

on  the  issue 

point  from 

answer 

Should  focus  on  answering  the  question  that  is  asked 

answering 

question 

question 

P7 

Must  know  audience 

Answered 

- 

Decision  makers  are  looking  for  answers,  not  just  facts 

question  more 

relating  to  the  question 

throughly 

P8 

Form  is  key  in  influencing  policy  decisions 

Was  not  clear 
it  answered 
question 

Table  8.  Summary  of  Responses  to  Comparison  of  Reports  Prompt,  Tone  of  Reports  Probe, 
commenting  on  Organization  and  Form 
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The  majority  of  feedback  about  the  organization  and  form  of  the  report  was  not  focused 
specifically  at  either  of  the  analysis  reports  within  the  scenario.  Rather,  participants  made 
general  prescriptive  comments  defining  appropriate  briefing  form.  What  few  report-specific 
comments  there  were  tended  to  focus  on  whether  or  not  the  report  was  successful  in 
explicitly  answering  the  task  question. 

As  a  related  observation,  note  that  P3  did  not  make  any  significant  comments  relative  to 
organization  and  form  aspects  of  the  tone  of  reports  probe.  This  fact  was  potentially 
attributable  to  limited  professional  experience,  given  his  status  as  the  most  novice 
participant  analyst  in  the  study. 

Every  participant  attended  to  at  least  one  of  the  three  components  of  the  tone  of  reports 
probe  in  comparing  the  briefing  reports.  Further,  with  some  exception  in  writing  style 
aspects,  participants  made  many  similar  comments  in  comparing  and  contrasting  the 
reports.  An  analysis  of  the  obstacle  sets  probe  also  identified  comparable  patterns  of 
response. 


"Presumably  the  author  of  report  two  also  came  across ,  I  would  hope,  some 
of  the  obstacles  that  are  listed  in  paper  one.  They  would  have  seen  [those 
obstacles]  in  the  course  of  their  analysis. " 

Although  similar,  participant  responses  to  the  obstacle  sets  probe  were  not  as  numerous  as 
they  were  to  tone  of  reports  probe.  As  shown  in  Table  9,  only  four  participants  made 
comments  indicating  that  they  recognized  the  set  of  obstacles  cited  in  the  briefing  reports 
were  incomplete.  P3  and  P5  noticed  that  the  reports  contained  different  obstacle  sets,  while 
PO  and  P6  identified  obstacles  of  interest  that  were  not  mentioned  in  either  report. 

In  contrast,  some  participants  responded  to  the  obstacle  sets  from  a  different  perspective. 
Three  participants  concluded  that  the  obstacles  identified  in  at  least  one  of  the  reports  were 
incorrectly  classified.  P2  and  P8  noted  that  the  A(1)  report  misclassified  risks  as  obstacles, 
while  P2  and  P4  judged  that  the  A(2)  report  focused  on  benefits  rather  than  obstacles. 

Two  participants,  PI  and  P7,  did  not  comment  on  the  obstacle  sets  at  all.  As  implied  by  the 
above  quotation,  this  could  potentially  be  due  to  the  participants  assuming  that  both 
analysts  in  the  scenario  encountered  the  same  obstacles  during  their  research,  then  further 
assuming  that  the  analysts  had  justification  for  identifying  the  obstacles  listed  in  their 
report  as  being  those  most  primary. 

The  obstacles  probe,  of  all  the  probes  in  the  comparison  of  reports,  generated  the  least 
response  from  participants  overall — perhaps  for  the  same  reason  noted  above.  Another 
possibility  is  that,  given  a  scenario  topic  was  designed  to  be  outside  their  areas  of  expertise, 
the  participants  were  less  inclined  to  respond  to  this  aspect  of  the  briefing  reports. 
Regardless,  the  responses  that  were  given  appeared  to  fit  the  pattern  of  similar  responses 
shown  in  the  tone  of  reports  probe  and  the  inclusion  of  figure  probe. 

"A  graphic  can  be  wonderful  and  tell  you  a  lot. " 
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ID 

Comments  Indicating  Incomplete  Set 

Comments  Indicating  Misclassified  Obstacles 

PO 

Neither  report  identified  the  counter¬ 
positions 

Neither  hit  on  the  alternatives  to  LNG 

A(2)  report  did  not  mention  counter¬ 
terrorism 

PI 

- 

- 

P2 

A(1)  report  must  be  clear  about  obstacles  vs. 
risks  vs.  downsides,  risk  is  not  an  obstacle 

A(2)  report  is  more  benefits  focused 

P3 

A(2)  report  has  info  about  environmental 
groups  being  in  disarray  not  in  A(1)  report 

- 

P4 

Too  much  about  benefits,  rather  than 
obstacles 

Too  much  price  focus,  price  is  not  an  obstacle 

P5 

Reports  indicate  different  obstacles 

A(2)  report  does  not  mention  terrorism  at 
all 

P6 

Energy  sector  is  an  obstacle  not  mentioned 
Both  ignored  key  barrier  of  infrastructure 

- 

P7 

- 

- 

P8 

- 

In  A(1)  report  some  obstacles  are  more  like 
risks 

Table  9.  Summary  of  Responses  to  Comparison  of  Reports  Prompt,  Obstacle  Set  Probe 
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The  final  probe  embedded  in  the  comparison  of  reports  prompt  was  the  inclusion  of  figure 
probe.  This  probe,  as  with  two  previously  described,  showed  a  consistency  of  response  across 
participants.  In  fact,  nearly  all  participants  indicated  that  they  found  the  inclusion  of  a  visual 
element  to  be  a  valuable  part  of  developing  an  effective  analysis  briefing.  However,  even  in 
acknowledging  the  value  of  the  graphic,  most  participants  recognized  that  the  content  of 
the  graph  did  not  relate  directly  to  the  scenario  task  question,  making  comments  such  as: 

"This  one  looked  a  little  more  rigorous.  They've  got  some  charts  here, 
which  they  obviously  got  from  the  internet  or  whatever. " 

"I  like  the  picture...  [It]  has  a  poor  graphic,  but  it  draws  me  there. " 

"The  chart  helped  focus  the  ideas,  but  it  wasn't...  labeled  as  far  as  obstacles. " 

"I  think  they  pulled  a  nice  graph.  And  so  they  were  trying  to  at  least  create  the 

illusion...  that  there  was  scientific  evidence  and  research  into  the  actual  data. " 

"Then  again,  what's  the  point  of  this  natural  gas  price  chart  in  relation  to  the 
question  that's  being  asked  here?  It  has  nothing  to  do  with  it. " 

"The  use  of  the  chart  is  gratuitous  and  does  not  even  reflect  the 
date  range  that  they  cite  in  the  paragraph. " 

Thus,  although  the  figure  was  seen  as  an  important  cue,  the  relationship  between  the 
inclusion  of  the  visual  and  inferences  that  could  be  made  about  rigor  were  not  as  clear.  Even 
so,  the  inclusion  of  figure  probe  did  generate  a  strong  response  from  participants. 

Across  the  probes  of  the  comparison  of  reports  prompt,  a  similarity  among  participant 
responses  was  observed.  Inclusion  of  figure  and  tone  of  reports  seemed  to  show  the  most 
similarity,  while  comments  regarding  the  differences  between  obstacle  sets  were  more 
diverse.  Generally,  the  responses  were  similar  in  that  participants  attended  to  similar  cues  in 
forming  their  assessments  of  the  quality  of  the  briefing  reports.  The  caveat  to  this  general 
finding,  however,  was  that  although  many  of  the  same  cues  were  attended  to  in  assessing 
the  analytic  products,  the  ways  in  which  those  cues  were  interpreted  by  participants  as 
relating  to  analytic  rigor  tended  to  be  quite  varied. 

3.6.2  Comparison  of  Process  Documents  Prompt 

The  second  discussion  prompt  in  the  study  focused  on  participants'  perceptions  of  the 
process  documents  as  valuable  in  revealing  the  processes  behind  the  analysis  briefing 
reports.  In  forming  this  assessment,  participants  used  the  available  process  documents  to 
build  an  understanding  of  the  analysis  process.  Each  of  the  seven  types  of  process 
documents  also  served  as  a  probe  in  the  sense  that  participants  attended  to  and  commented 
on  some  of  the  documents,  while  minimizing  or  disregarding  others.  The  general  finding  of 
the  comparison  of  process  documents  prompt  was  that,  while  participants  valued  the 
process  documents  similarly  when  ranking  them,  there  were  notable  differences  in  how  the 
documents  were  used  to  judge  rigor. 


33 


“It  helped  a  lot  to  [see  the  process  documents]  because... 
it  confirms  what  I  thought  about  the  priority  of  these  issues. 


In  comparing  the  analysis  processes,  participants  ranked  the  usefulness  of  each  of  the 
different  process  documents  in  making  assessments  of  process  rigor,  assigning  a  rank  of  one 
to  the  most  useful  process  document  and  a  rank  of  seven  to  the  least  useful.  Kendall's 
coefficient  of  concordance  (1/1/  =  0.526,  p  <  0.01)  indicated  there  was  agreement  among 
participants  as  to  the  order  in  which  the  process  documents  were  valuable.  The  composite 
rankings  of  these  documents,  found  in  Table  1 0,  point  toward  facets  of  the  analytic  process 
that  are  perceived  as  most  critical  in  supporting  an  understanding  of  rigor. 

Paired  Wilcoxon  Rank-Sum  Tests  (p  <  0.01)  order  the  data  into  three  tiers  of  importance  in 
judging  rigor,  with  the  first  tier  reflecting  those  most  useful  in  assessing  rigor  and  the  third 
tier  reflecting  those  least  useful.  Of  the  process  documents  available  in  the  study,  all  but  one 
participant  ranked  "Key  Documents"  and  "Hypotheses  Considered"  as  two  of  the  top  three 
most  valuable  sources  of  process  information. 

In  contrast,  all  but  one  participant  ranked  "Where  Documents  Stored"  as  the  least  valuable. 
The  four  other  types  of  process  documents  fell  into  an  intermediate  category,  with  some 
participants  valuing  them  highly  and  others  seeing  them  as  minimally  valuable.  This  data 
implies  that,  for  some  types  of  process  information,  there  was  a  tacitly  consistent  distinction 
among  participants  as  to  which  information  sources  were  perceived  as  being  meaningfully 
diagnostic  of  process  rigor  and  which  were  judged  as  conveying  little  meaning. 


Document 

Tier 

Mean 

Range 

P0 

PI 

P2 

P3 

P4a 

P5 

P6b 

P7C 

P8d 

Key 

Documents 

First 

2.00 

1-3 

3 

2 

1 

1 

2 

3 

3 

1 

2 

Hypotheses 

Considered 

2.22 

1-7 

1 

1 

7 

2 

1 

1 

2 

2 

3 

Collaborations 

Second 

4.00 

2-5 

2 

3 

5 

4 

5 

4 

5 

4 

4 

Research 

Note-Sheet 

4.06 

1-6 

6 

4 

2 

6 

6.5 

5 

1 

5 

1 

Documents 

Read 

4.17 

2-7 

5 

6 

3 

3 

3 

2 

6 

3 

6.5 

Query 

Summary 

5.00 

4-6 

4 

5 

6 

5 

4 

6 

4 

6 

5 

Where 

Documents 

Stored 

Third 

6.56 

4-7 

7 

7 

4 

7 

6.5 

7 

7 

7 

6.5 

Note.  Rankings  were  standardized  for  participants  whose  rating  method  deviated  from  a  1-7  scale  as  noted. 
Participant  did  not  rank  "Research  Note-Sheet"  or  "Where  Documents  Stored",  noting  that  neither  was 
perceived  as  valuable.  Participant  originally  ranked  "Research  Note-Sheet"  as  1  and  "Hypotheses  Considered" 
as  1A.  Participant  added  an  item  to  the  list  of  process  documents,  ranking  "Sources  of  Key  Documents"  after 
"Key  Documents"  but  before  "Hypotheses  Considered."  Participant  did  not  rank  "Documents  Read"  or 
"Where  Documents  Stored",  noting  that  neither  was  perceived  as  valuable. 

Table  10.  Process  Document  Rankings 
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In  addition  to  analyzing  the  process  document  comparisons  by  contrasting  the  rankings, 
another  perspective  on  the  data  is  gained  in  viewing  the  process  documents  as  embedded 
probes.  Insight  into  how  participants  transformed  the  partial  information  provided  in  the 
process  documents  into  a  more  complete  assessment  of  rigor  is  gained  by  organizing  the 
participant  feedback  relative  to  these  seven  probes.  This  approach  affords  the  exploration 
of  the  ways  in  which  the  process  documents  were  used  by  participants  to  assess  rigor. 

"If  [the  analyst]  did  the  research  and  came  up  with  the  conclusion  that  [a  certain 
obstacle]  is  really  not  that  big  of  a  deal  based  on  the  research,  therefore  [it  was] 
dropped...  that's  a  much  more  rigorous  process  as  opposed  to  the  analyst  [who] 
dropped  it  because  he  didn 't  do  his  homework. " 

While  there  was  some  agreement  among  participants  about  the  order  in  which  process 
documents  were  useful,  there  were  differences  in  what  those  process  information  cues 
indicated  to  each  participant  about  rigor.  Insights  from  participants  regarding  the  value  of 
each  type  of  process  document  revealed  that  they  were  viewed  as  providing  both  positive 
and  negative  diagnostic  information  about  analytic  rigor.  Examples  include: 

"Hypotheses...  show  that  the  analyst  looked  for  alternatives." 

"I  would  want  to  know  what  hypotheses  did  they  have  going  in  or  formulate  while 
they  were  working.  It's  trying  to  get  into  their  mindset  as  they  did  their  work  to  try 
to  determine  if  that...  biased  what  they  paid  attention  to  or  didn 't  pay  attention  to. " 

"Collaboration...  gives  you  more,  showing  that  the  person  did  his  homework. " 

"The  fact  that  the  analyst...  discussed  an  initial  search  with  somebody  and  that  was 
it  [shows]  that  he  didn 't  have  a  grasp  of  the  scope  of  the  problem. " 

"I  found  the  formal  notes  to  be  useful.  They  were  helpful  in  understanding  how 
the  analyst  went  amiss...  [and]  gave  me  insight  into  the  recovery  process. " 

"The  fact  that  they  didn't  use  a  note-sheet  speaks  to  the  fact  that 
their  presentation  wasn't  as  good  as  it  could  have  been. " 

Across  types  of  process  documents,  the  data  show  the  same  pattern  of  both  positive  and 
negative  inferences  made  about  process  rigor  based  on  the  identical  cues.  That  is,  the  data 
indicate  that  information  in  the  process  documents  was  valuable  to  participants  in  different 
ways.  This  finding  parallels  that  indicated  by  the  first  discussion  prompt  in  investigating 
assessments  made  based  on  product — while  there  was  general  agreement  about  which  cues 
matter  in  making  assessments  of  rigor,  there  were  differences  among  participants  in  how 
those  cues  were  transformed  into  overall  judgments  of  analytic  rigor. 

3.6.3  Rigor  Assessment  Prompt 

The  first  decision  prompt  participants  addressed  was  rigor  assessment,  as  they  were  twice 
asked  to  judge  whether  A(1)  or  A(2)  was  more  rigorous.  This  decision  was  made  both  after 
comparing  the  briefing  reports  and  after  comparing  the  process  documents.  The  general 
finding  from  comparing  the  participants'  responses  was  that  gaining  insight  into  the 
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analysis  process  changed  perceptions  of  analytic  rigor — even  in  participants  who 
commented  that  it  should  not  or  would  not  do  so.  This  prompt  also  examined  the 
approaches  participants  preferred  to  use  to  discover  the  rigor  of  an  analysis  process. 

"I  try  to  stress  to  the  people  I  work  with  that  the  process  does  not  have  to  be  rigid. 

You  don't  have  to  do  A  then  B  then  C.  You  can  do  A  then  F  then  B  then  G. " 

Table  1 1  shows  participant  responses  both  before  and  after  reviewing  the  process 
documents.  In  assessing  rigor,  five  of  nine  participants,  after  reviewing  the  process 
documents,  altered  their  assessments  of  whether  they  perceived  A(1)  or  A(2)  as  being  more 
rigorous.  Additionally,  the  data  indicted  that,  despite  the  study  design,  neither  analysis 
process  was  clearly  more  rigorous  than  the  other,  as  two  of  nine  participants  identified  A(2) 
as  being  more  rigorous,  even  after  seeing  the  process  documents. 

The  table  also  gives  brief  descriptions  of  the  primary  reasons  participants  offered  in  making 
their  original  assessments  of  rigor  and  for  revising  their  assessments  of  rigor.  This  data 
provides  insight  into  what,  at  a  glance,  participants  deemed  as  being  most  salient  in  judging 
rigor.  A  related  finding,  based  on  similar  comments,  was  that  the  introduction  of  process 
insight  seemed  to  influence  perceptions  of  rigor  even  in  participants  who  indicated  that  it 
did  not,  would  not,  or  should  not,  significantly  influence  their  assessment.  For  example: 

"The  process  documents  lined  up  with  my  expectations. " 

"My  opinion  didn 't  really  change  of  the  quality  of  the  product. 

It  confirmed  what  I  thought. " 

Of  the  analysts  who  made  the  comments  such  as  these,  half  actually  did  changed  their 
assessment  of  which  process  was  perceived  as  more  rigorous  after  reviewing  the  process 
documents.  The  insight,  then,  is  that  access  to  analysis  process  information  is,  potentially, 
more  powerful  in  shaping  perceptions  of  rigor  than  many  of  the  participants  realized. 

"Rigor  is  fleshed  out  in  the  review  process.  If  there  is  a  problem  then  we  go  deeper. " 

As  part  of  the  rigor  assessment  prompt,  participants  were  also  asked  to  describe  the  process 
they  would  use  to  discover  the  rigor  of  an  analysis  process,  given  unrestricted  access  to  the 
resources  they  might  request.  This  question  was  posed  to  analysts  before  they  reviewed  the 
process  documents.  As  shown  in  Table  12,  responses  to  this  inquiry  tended  to  fall  into  three 
categories:  analyst  interaction,  analyst  background,  and  process  dimensions. 

Analyst  interaction  describes  comments  from  participants  that  indicated  they  would  use  a 
form  analyst-to-analyst  interaction  to  discover  process  rigor.  Analyst  background  covers 
comments  from  participants  indicating  that  they  would  want  to  know  background 
information  about  the  analyst  who  produced  the  briefing.  Finally,  process  dimensions 
identify  the  aspects  of  the  analytic  process  that  participants  would  be  most  interested  in 
understanding  when  judging  rigor.  As  noted  earlier,  these  perspectives  were  voiced  before 
the  analyst  reviewed,  or  were  made  aware  of,  the  available  scenario  process  documents. 


36 


ID  Bfr.  Rationale 


Aft.  Rationale 


Chg.  Rationale 


PO  A(2) 


Tailored  for  policy¬ 
makers,  looked  at 
context  issues, 
explored  the  issues 
more 

Confident  in  seeing  a 
more  extensive 
bibliography 


A(1)  Better  rigor,  but  does  not 
make  final  report  any 
better,  prefers  A(2)  report 

Information  heavy.  Data  is 
there,  but  not  in  proper 
format 

Analyst  afraid  of  picking 
and  choosing  information 


Yes  A(2)  had  enough 
sources,  but  seeing 
process  documents 
added  more  questions 


PI 

A(1) 

Good  preciseness, 
objectivity,  lack  of 
bias 

A(2) 

Saw  more  rigor  than  was 
assumed,  shows  good 
collaboration,  feels  a  little 
better  about  conclusions 

Yes  A(1)  had  impressive 
number  of 
documents,  but  they 
are  still  insufficient 

Confounded  by  thinness  of 
documents  available,  still 
needs  something  behind  it 

More  reluctant  to 
forward  report,  would 
tell  analyst  to  expand 

P2 

A(2) 

Higher  quality  shows 
more  rigor,  provided 
more  information 

A(1) 

Better  rigor,  but  still  not 
good  enough 

Yes  A(2)  did  not  have 

enough,  would  have 
liked  to  see  more 

P3 

A(1) 

Addressed  broader 
concerns 

A(1) 

Quality  is  better  because 
quantity  is  better 

- 

P4  A(1)  Seems  to  have 

greater  relative  rigor 
in  comparison,  but 
still  a  low  "absolute" 
rigor 


P5  A(1) 


Not  as  much  opinion, 
indicates  good 
research 


But,  terrorism  focus  implies 
analyst  had  idea  going  in 
and  looked  for  confirmation 

A(1)  Research  documents 

indicated  more  rigor  than 
report  reflected,  problems 
in  sourcing  and  answering 
question 

A(1)  Good  documentation  gave 
confidence  in  process, 
reflected  a  more  rigorous 
process 

Didn't  have  as  many  doubts, 
had  more  stylistic  questions 


P6 

A(1) 

Evidence  of  research, 
indicates  more 
organized  process 

A(1) 

Documentation  implied 
more  rigor,  which  was 
reflected  in  report 

P7 

A(1) 

- 

A(2) 

More  balanced,  better 

Yes  A(1)  has  good 

P8  A(2)  Tighter  form, 

tailored  to  decision 
maker,  included  chart 


A(1) 


research  on  topic,  needs  to 
work  on  presentation.  Did 
not  really  collaborate 

Good  depth  of  research, 
obstacles  focused, 
organized 


Yes 


presentation,  but 
poor  research,  heavily 
biased  toward 
terrorism  issue 

A(2)  style  was  good, 
but  research  was 
lacking 


Note.  "Bfr."  lists  the  analytic  process  identified  by  participants  as  appearing  more  rigorous  before  viewing  the 
process  documents,  while  "Aft."  lists  the  process  they  identified  after  viewing  the  process  documents.  "Chg." 
refers  to  whether  or  not  the  participant  changed  their  assessment  of  rigor  after  viewing  the  process  documents. 


Table  1 1.  Assessment  of  Rigor  Before  and  After  Viewing  Process  Documents 
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ID  Analyst  Interactions 


Analyst  Background  Process  Dimensions 


PO  Would  meet  with  author  and 
specify  problems,  peer  review 


PI  Would  ask  questions  in  course 
of  day-to-day  work,  responses 
point  to  areas  that  need  work 


Outlines  of  report 
Hypotheses 
Collaboration 
Sources 

Client  requirements 

Looking  for  varied  levels  of 
detail  in  understanding 

Sources,  foundational 
documents,  background 
information 


P2  Discussion  with  analyst,  ask 
them  what  they  did  it 

Would  start  with  the  task 
question,  then  go  to  "second 
level"  questions 


Would  want  to  know  analyst 

Wants  analyst  to  "own"  their 
analysis  processes,  do  they 
believe  these  things? 


Looking  for  thinking  from  a 
multitude  of  perspectives 

Justifications  and  key 
indicators 

Sources 


P3  Directly  ask  analyst,  "how  did 
you  come  to  these 
conclusions?" 


Credibility  of  sources 

Due  diligence,  source  of 
sources 

Collaboration 


P4  Face  to  face  meeting  or 

sending  notes  with  questions 

General  collaborative  peer 
review  process 


P5 


P6 


Might  be  interested  in 
analysts  background,  but 
maybe  not 

Would  want  access  to  their 
past  work,  is  it  improving? 


Sources 

Use  of  connections 
Stated  confidence 

Evidence  of  thinking  about 
"next  step"  obstacles 

Sources,  public  vs.  classified 


Sources 

Research 

Collaboration 


P7  Direct  collaboration 


P8  Asking  questions, 

communication  via  e-mail 


Knowledge  of  analyst 
background  and 
qualifications 


Sources,  how  current  are 
they? 

Source  Diversity,  verification 
that  they  cover  all  places  they 
are  likely  to  find  information 

Sources 


Table  12.  Participant  Preferred  Approaches  to  Discovering  Process  Rigor 
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The  general  theme  across  comments  about  analyst  interactions  was  that  interpersonal 
feedback  was  viewed  as  a  critical  way,  if  not  the  way,  participants  preferred  to  discover  the 
rigor  behind  an  analysis  process.  Regarding  process  dimensions,  every  participant  indicated 
a  preference  for  seeing  original  source  information.  Collaboration  was  mentioned  by  a 
number  of  participants  as  well.  However,  beyond  that,  participants  voiced  many  variants  on 
the  information  to  which  they  would  like  to  have  access  when  making  a  judgment  of  rigor. 
There  were  also  mixed  results  as  to  whether  or  not  knowing  an  analyst's  background  would 
be  of  value,  with  some  citing  it  as  important  to  know  and  others  expressing  ambivalence. 

Although  these  comments  were  valuable,  the  overarching  message  of  the  rigor  assessment 
prompt  is  that  process  insight  can  influence  perceptions  of  rigor.  This  finding  was  observed 
even  in  participants  who  did  not  expect  the  information  to  influence  their  judgments  and 
even  though  participants  were  given  access  only  to  limited,  incomplete  information  about 
process  in  the  form  of  process  documents.  This  finding  was  similarly  observed  in  the  second 
decision  prompt,  the  Supervisor's  Dilemma. 

3.6.4  Supervisor's  Dilemma  Prompt 

The  Supervisor's  Dilemma  prompted  participants  to  decide  whether  or  not  the  analysis  reports 
in  the  scenario  were  ready  to  be  sent  to  a  decision  maker.  As  in  the  rigor  assessment 
prompt,  participants  made  this  decision  both  before  and  after  viewing  the  process 
documents.  Also  similar  to  the  rigor  assessment  prompt,  the  Supervisor's  Dilemma  found  that 
insight  into  process  produced  a  change  in  assessment,  as  revealed  in  Table  13.  In  this  case, 
the  observed  change  was  in  regard  to  the  assessment  of  quality  of  the  analytic  product, 
rather  than  process. 


Rigor 

Assessment 

Supervisor's  Dilemma 

ID 

Order 

Before 

After 

Before 

After 

Comments  Before  Viewing  Process  Documents 

PO 

A(1) 

A(2) 

A(1) 

A(2) 

A(2) 

A(2)  report  might  be  ready,  needs  "action  items"  first 

PI 

A(1) 

A(1) 

A(2) 

A(1) 

NR 

A(1)  report  is  ready  for  policy-maker 

P2 

A(1) 

A(2) 

A(1) 

NR 

NR 

A(2)  report  is  better,  but  neither  is  rigorous  enough 

P3 

A(2) 

A(1) 

A(1) 

NR 

NR 

Both  reports  need  editing,  hard  to  tell  which  is  better 

P4 

A(1) 

A(1) 

A(1) 

NR 

NR 

A(1)  report  is  better,  attempts  to  look  at  issues 

P5 

A(2) 

A(1) 

A(1) 

NR 

NR 

A(1)  report  is  better,  but  would  still  require  edits 

P6 

A(2) 

A(1) 

A(1) 

NR 

NR 

A(1)  report  was  much  better,  but  still  not  ready  yet 

P7 

A(2) 

A(1) 

A(2) 

A(1) 

NR 

A(1)  report  isn't  perfect,  but  might  be  good  enough 

P8 

A(1) 

A(2) 

A(1) 

NR 

NR 

A(2)  report  closer  to  being  ready 

Note.  A  mark  of  "NR"  reflects  a  participant  who  indicated  that  neither  report  was  ready  to  send  forward. 


Table  13.  Run  Order  and  Rigor  Assessment  and  Supervisor's  Dilemma  Probes 
Before  and  After  Viewing  Process  Documents,  Including  Comments 
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Unlike  the  assessment  of  rigor  prompt,  however,  the  Supervisor's  Dilemma  had  relatively  few 
participants  identify  either  of  the  reports  as  ready  to  send  to  a  decision  maker — a  surprising 
result  potentially  attributable  to  the  fact  that,  in  this  prompt,  participants  were  asked  for  an 
absolute  judgement  of  acceptability,  rather  than  a  relative  judgement  of  quality. 

Before  seeing  the  process  documents  only  three  of  the  nine  analysts  indicated  that  either  of 
the  reports  was  ready  for  a  decision  maker.  After  seeing  the  process,  only  one  thought  that 
either  report  was  acceptable  for  a  decision  maker.  Interestingly,  it  was  the  A(2)  briefing 
report  that,  after  the  review  of  the  process  documents,  was  the  only  one  deemed  to  be 
acceptable.  No  participant  indicated  that  both  briefings  were  ready.  Consequently,  the  data 
show  that  two  of  the  participants  altered  their  assessments  as  a  result  of  insight  into  the 
analysis  process. 

"It  seems  to  me  there's  more  conjecture...  in  this  [briefing  report].  And  so  I  would 
have  to  question:  Am  I  reading  opinions  or  am  I  reading  facts?  As  opposed  to  this 
[other]  one,  I  get  more  of  a  'warm  and  fuzzy'  out  of.  I'm  not  seeing  opinions  written 
in  here.  There 's  more  to  [analysis]  work  than  providing  a  report  to  a  policy  maker. " 

Also  shown  in  Table  13,  the  data  did  not  indicate  a  strong  relationship  between  the  order  in 
which  the  analyses  were  labeled  for  participants  and  the  likelihood  that  they  preferred  one 
process  over  the  other.  While  there  was  a  slight  tendency  for  participants  to  view  the 
analysis  process  of  the  second  report  that  as  being  more  rigorous,  at  least  in  initial 
assessment,  this  minor  relation  was  not  seen  as  meaningful  in  the  context  of  the  study.  Thus, 
as  expected,  the  order  in  which  the  reports  were  labeled  was  not  a  major  factor  in 
influencing  participants'  decisions  of  report  readiness  for  a  policy  maker. 

Process  insight,  it  seems,  does  have  the  potential  to  influence  the  ambiguous  "warm  and 
fuzzy"  alluded  to  in  the  above  quotation.  This  finding  even  seems  to  hold  when,  as 
instanced  in  the  Supervisor's  Dilemma,  there  is  not  a  clear  consensus  as  to  which  report  is 
better.  Taken  together,  the  data  from  the  decision  prompts  indicate  that  providing  insight 
into  an  analysis  process  influences  assessments  of  both  analytical  rigor  and  product  quality. 


3.7  Participant  Insights 

In  addition  to  the  structured  feedback  provided  through  the  decision  and  discussion 
prompts,  participants  offered  many  other  insights  throughout  the  elicitation  sessions  that 
were  not  directly  related  to  the  prompts.  In  particular,  during  the  debrief  sessions, 
participants  were  asked  to  identify  what  they  saw  as  the  most  significant  challenge 
currently  facing  the  intelligence  community.  As  seen  in  Table  14,  there  were  a  wide  variety 
or  responses  to  this  question. 

"[Policy  makers]  are  too  far  removed  from  the  analysts  that  actually  do  the  work...  If 
there  are  questions  or  clarifications  that  are  necessary  they  may  or  may  not  filter 
back  down  to  the  person  who  actually  did  the  work.  [Instead]they  may  be  answered 
by  some  intermediary  who  really  doesn 't  have  the  expertise  to  be  answering  the 
question,  but  think  they  do  because  they're  the  ones  that  conveyed  the  report. " 
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ID  Challenge 


Description 


PO  Producing  actionable 

intelligence,  given  constraints 


PI  Lack  of  data  visualization  aids 

P2  "Good  news  doesn't  sell" 


P3  Re-syndication  of  information 


P4  Policy  makers  see  what  they 
want  to  see 


P5  Analysis  could  be  taken  out  of 
context 


There  is  never  enough  time  to  work  a  problem,  often 
policy-makers  "want  it  yesterday"  and  want  information 
"now  or  never",  even  for  big  complex  questions. 

This  requires  a  dynamic  process.  What  happens  if  process 
time  is  shortened?  Is  there  a  way  to  short-circuit  the  process 
while  still  maintaining  rigor? 

Lack  of  stock  tools  that  grab  disparate  data  and  collect  it 
into  meaningful  visuals. 

Nobody  wants  to  hear  good  things.  But,  when  you  look  for 
bad  things,  everything  looks  bad. 

We  are  here  to  warn  people,  not  give  the  "warm  and 
fuzzies." 

Citing  of  sources  is  an  emerging  global  issue.  Things  just 
seem  to  emerge.  "I  read  it  so  it  has  to  be  true." 

Proponent  of  group  documentation  for  facts  and  history, 
e.g.,  Wikipedia.  Allows  for  multiple  contributing 
perspectives. 

Difficult  for  policy  makers  to  overcome  their  preconceived 
notions.  They  can  ask  for  additional  analysis  until 
supporting  information  is  found. 

Analysis  could  be  misunderstood  or  misused  by  itself. 

Should  be  taken  in  combination  with  other  INTS. 


P6  Pressure  to  please  customer  There  is  a  pressure  to  give  the  right  answer.  There  often  is 

no  "right"  answer.  But,  making  policies  requires  a  right 
answer. 

If  analysis  process  is  influenced  too  much,  there  is  potential 
for  bad  decisions.  Policy  makers  may  keep  asking  until  they 
get  the  answer  they  want. 

Policy  makers  assume  reports  come  from  "the  expert." 
Really  they  are  just  getting  "best  guesses." 


P7  Policy  makers  are  too  far  Disconnect  in  producer  and  consumer  relation.  Person  who 

removed  from  people  doing  the  did  work  should  always  be  included  in  work  that  goes 

actual  work  forward. 

There  are  barriers  to  getting  back  to  the  source.  If  a 
question  or  clarification  is  necessary,  it  may  not  filter  back 
to  original  author. 

Policy  makers  are  too  busy,  or  it  is  "beneath"  them  to  go  to 
the  source.  Typically  questions  are  answered  by 
intermediaries.  They  hear  frequently  from  people  they  are 
comfortable  with. 


P8  Getting  info  to  policy  maker 
about  obstacles  encountered  in 
doing  analyses 


E.g.,  getting  across  what  was  not  covered  in  an  analysis. 

There  is  only  so  much  information  analysts  have  access  to, 
given  constraints. 


Table  14.  Participant  Identified  Primary  Challenge  Facing  the  Intelligence  Community 
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Admittedly,  given  limited  time  to  consider  the  question,  it  is  unlikely  that  participants 
identified  as  the  greatest  challenge  in  intelligence  what,  upon  further  reflection,  they 
would  have  named  as  being  most  important.  Rather,  most  participants  probably  identified  a 
significant,  or  salient,  challenge  to  the  analysis  community  relevant  to  the  earlier 
discussions.  Even  so,  participants  generated  many  interesting  observations  that  reflect  some 
of  their  current  concerns  as  intelligence  analysts. 

A  common  theme  observed  throughout  those  comments  was  the  identification  of 
challenges  that  arise  during  the  transfer  of  an  analysis  product  from  analyst,  as  producer,  to 
policy  maker,  as  consumer.  In  fact,  PO,  P4,  P5,  P6,  P7,  and  P8  each  identified  a  challenge 
related  to  the  difficulties  in  the  interactions  between  analyst  and  policy  maker — a  finding 
that,  it  might  be  argued,  directly  supports  the  value  of  better  understanding  the  role  of 
rigor  in  intelligence  analysis. 

Across  the  results  of  the  study,  two  general  patterns  emerged.  First,  the  data  support  the 
finding  that  providing  insight  into  the  analysis  process  produces  change  in  perceptions  of 
rigor.  Second,  the  data  also  indicate  that  there  was  variability  in  how  participants  converted 
cues  to  rigor  into  overall  assessments  of  rigor,  even  though  they  tended  to  use  similar  cues 
in  forming  their  judgments.  In  the  next  section,  the  implications  of  these  findings  are 
discussed  in  relation  to  the  broader  intelligence  and  information  analysis  contexts. 
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4.  IMPLICATIONS  OF  THE  FINDINGS  ON  RIGOR 


This  section  presents  a  discussion  of  the  contributions  and  findings  the  emerged  from  the 
study  of  professional  analysts  described  in  this  thesis.  In  addition,  it  explores  a  revised 
definition  of  analytical  rigor  in  intelligence  analysis,  also  commenting  upon  an  alternative 
perspective  toward  rigor  that  exists  within  the  intelligence  analysis  community.  Finally,  the 
section  includes  a  discussion  of  the  limitations  of  the  design  methodology  and  identifies 
future  research  directions  for  the  continued  exploration  of  the  concept  of  rigor  within  the 
contexts  of  intelligence  and  information  analysis. 


4.1  Methodological  Contributions 

The  development  of  the  LNG  Scenario  is  a  significant  product  of  this  research  in  that  it 
represents  a  refined  scenario  case  with  potential  applications  that  extend  beyond  the 
exploration  of  rigor  in  information  analysis  discussed  in  this  work.  As  a  scenario,  it  provides 
a  reusable  cognitive  case  whose  critical  dimensions  can  be  manipulated  and  reshuffled  in 
various  ways  to  study  different  themes  in  information  analysis.  In  the  LNG  Scenario,  the 
critical  dimensions  of  the  case  are  manifested  in  the  four  interacting  aspects  of  energy, 
siting,  safety,  and  security — abstracted  as  the  respective  cognitive  components  of 
replanning,  planning,  analysis,  and  reanalysis.  These  dimensions,  then,  serve  as  the  basic 
levers  that  can  be  manipulated  to  explore  other  facets  of  information  analysis  in  other 
contexts. 

The  conceptualization  of  Supervisor's  Dilemma  also  represents  an  important  contribution  of 
this  research  effort  by  highlighting  the  rigor  judgement  decision  that  occurs  at  the  interface 
between  analysis  and  replanning  in  information  analysis.  This  framing  avoids  some  of  the 
controversies  in  the  intelligence  community  surrounding  the  relationship  between  analysts 
and  decision  makers,  by  leveraging  the  supervisor  position  as  an  intermediate  perspective 
that  falls  somewhere  between  these  two  roles.  The  dilemma  is  primarily  valuable  though, 
because  it  operationalizes  a  basic  aspect  of  avoiding  shallow  analysis.  That  is,  realizing  when 
an  analysis  is  too  shallow — and  telling  when  more  needs  to  be  invested  in  the  analysis 
process — is  a  fundamental  part  of  what  it  means  cognitively  to  perform  information 
analysis.  The  Supervisor's  Dilemma  captures  this  critical  function  in  a  way  that  is  easily 
integrated  into  studies  of  rigor  in  information  analysis. 

Perhaps  most  importantly,  this  study  represents  a  validation  of  the  Elicitation  by  Critique 
approach  as  an  effective  method  for  generating  feedback  from  expert  practitioners  about 
their  work.  This  research  applied  the  EBC  method  with  a  different  study  design  and  with  a 
different  type  of  analyst,  yet  it  succeeded  in  eliciting  significant  amounts  of  data  in  a 
domain  that  is  highly  restricted  by  security  concerns.  Additionally,  because  participants 
critiqued  the  analysis  processes  of  other  analysts,  rather  than  performing  analyses 
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themselves,  the  format  of  the  study  was  sensitive  to  the  time  considerations  of  the 
participants  as  practicing  professionals.  Even  so,  and  in  spite  of  their  other  commitments,  all 
participants  were  quite  willing  to  commit  the  full  time  alloted  to  the  study — and  most 
probably  would  have  continued  sharing  insights  if  given  additional  time — further  attesting 
to  the  value  of  the  method. 

A  related  benefit  of  the  study  was  that,  as  a  modified  version  of  the  EBC  approach,  it 
explored  a  number  of  refinements  of  the  methodology.  Most  notably,  the  method  used  in 
this  study  incorporated  a  scenario  walkthrough  component  through  the  LNG  Scenario  and  a 
decision  prompt  component  through  the  Supervisor's  Dilemma.  The  result  was  a  study  format 
that  generated  both  verbal  and  behavioral  data.  The  modified  method  was  also  structured 
such  that  the  process  documents  were  revealed  midway  through  the  study,  effectively 
producing  a  partial  debrief  in  the  middle  of  each  elicitation  session.  These  changes 
ultimately  led  to  a  study  design  that  produced  a  more  rich  and  faceted  data  set  than  might 
have  been  collected  using  the  basic  EBC  method  alone.  Thus,  not  only  did  this  study  serve  to 
validate  the  overall  EBC  approach,  but  it  also  went  a  step  further  by  incorporating  potential 
— and,  based  on  the  study  findings,  ultimately  valuable — refinements  to  the  basic  method. 


4.2  A  Discussion  of  Findings 

The  results  of  the  study  are  discussed  relative  to  the  overarching  purposes  of  the  research  to 
refine  the  understanding  of  rigor  in  information  analysis  and  to  identify  leverage  points  for 
improving  the  connection  between  analysis  and  replanning  in  intelligence  analysis.  The 
results  are  discussed  first  in  relation  to  the  two  overarching  research  questions,  as  identified 
in  Section  2,  and  second  in  relation  to  the  broader  goals  of  expanding  and  supporting  the 
understanding  of  rigor  in  both  intelligence  and  information  analysis. 

4.2. 1  Impact  of  Process  Insight  on  Judgment  of  Analytic  Rigor 

The  basic  finding  of  the  study  regarding  the  primary  research  question  was  that  process 
insight  did,  in  fact,  influence  judgement  of  analytical  rigor.  The  study  found  that,  relative  to 
both  the  rigor  assessment  prompt  and  the  Supervisor's  Dilemma  prompt,  participants  altered 
their  perceptions  of  rigor  based  on  this  insight.  This  effect  was  observed  even  in  participants 
who  stated  that  process  insight  would  not  influence  their  judgement.  Moreover,  the  results 
indicate  that  even  partial,  incomplete,  and  second-hand  process  information  can  initiate 
change  in  assessments  of  rigor,  as  participants  in  the  study  made  their  judgments  of  rigor 
based  only  on  the  review  of  process  documents — and  not  on  a  direct  observation  of  the 
analytic  processes  or  on  any  direct  interaction  with  the  author  analysts. 

This  highlights  an  important  distinction  between  perceived  rigor,  based  on  cues  inferred 
from  an  analytic  product,  and  effective  rigor,  based  on  insight  into  an  analytic  process — 
indicating  that  they  often  may  not  be  aligned.  While  it  was  found  that  professional 
intelligence  analysts  often  made  insightful  assessments  of  perceived  rigor,  the  results  of  the 
study  indicate  that  these  perceptions  were  apt  to  change  with  the  addition  of  process 
insight.  This  finding  both  supports  and  extends  the  observation  of  Tufte  (2003)  that  the 
form  of  an  analysis  product — in  this  case  a  briefing  report,  rather  than  a  PowerPoint 
presentation — can  distort  the  understanding  of  the  process  the  produced  it. 
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There  are,  however,  two  related  caveats  to  these  study  findings  that  should  be  noted.  First, 
there  were  some  participants  who  were  reluctant  to  revise  their  assessments  of  rigor,  even 
in  the  face  of  seemingly  conflicting  data.  For  example,  PO  continued  to  judge  the  A(2) 
briefing  report  as  being  better,  even  in  acknowledging,  after  reviewing  the  process 
documents,  that  A(1)  was  the  more  rigorous  process.  This  behavior  is  not  entirely  surprising, 
however,  and  might  be  characterized  as  a  manifestation  of  fixation,  as  defined  in  the 
psychology  literature,  in  that  participants  hypothesized  that  their  perceived  rigor  judgments 
were  accurate,  even  when  new  data  suggested  that  they  were  not  (Fraser,  Smith,  &  Smith, 
1992).  This  behavior  might  alternatively  be  viewed  as  resulting  from  the  utilization  of 
knowledge  shield — in  selectively  attending  only  to  the  information  that  supports  an  initial 
hypothesis,  for  example  (Feltovich,  Coulsen,  Spiro,  &  Adami,  1994). 

The  second  caveat  was  the  tendency  for  some  participants  to  interpret  weaknesses  indicated 
by  the  process  documents  as  being  indicative  of  deficiencies  in  the  abilities  of  the  author 
analyst,  rather  than  as  deficiencies  of  analysis  process.  P3,  for  instance,  commented  that 
both  reports  reflected  an  unseasoned  analyst.  The  emergence  of  this  inclination  in  some 
participants  was  consistent  with  the  research  on  attribution  error,  given  that  the  cues 
available  in  the  process  documents  were  inherently  ambiguous  and  could  feasibly 
interpreted  multiple  ways  (Ross  &  Anderson, 1982;  Fraser,  Smith,  &  Smith,  1992).  Also 
contributing  to  the  emergence  of  this  behavior,  at  least  in  part,  was  the  fact  that 
participants  were  told  at  the  outset  of  the  study  that  they  were  critiquing  the  analysis 
processes  of  junior  analysts — likely  increasing  the  chance  that  they  would  made  a 
dispositional,  rather  than  situational,  attribution  of  process  weaknesses. 

The  overall  message,  then,  is  that  simply  providing  process  information  does  not  guarantee 
a  change  in  perspective.  Rather,  it  is  important  to  recognize  that  certain  types  of  process 
information  facilitate  the  realization  of  process  insight,  different  types  of  which  may  be 
better  or  worse  in  influencing  revisions  of  perceived  rigor.  While  this  study  does  not  purport 
to  define  what  is  precisely  the  right  process  information  to  provide,  the  findings  related  to 
the  second  research  question  do  point  toward  the  facets  of  analytic  process  that  are  most 
critical  in  making  judgments  of  rigor. 

4.2.2  Cues  for  Inferring  Rigor  in  Intelligence  Analysis 

The  study  data  indicate  that,  while  many  of  the  same  cues  were  used  to  make  inferences 
about  analytic  rigor,  there  were  substantial  differences  among  participants  as  to  which  cues 
were  perceived  as  being  most  relevant  and  in  how  judgments  of  rigor  were  drawn  from 
these  cues.  For  example,  most  participants  tended  to  rank  collaboration  as  an  important  cue 
for  assessing  rigor  in  an  analysis  process.  Flowever,  some  participants  viewed  it  as  a  positive 
indicator  of  analytic  rigor  (e.g.,  as  indicating  that  multiple  perspectives  were  incorporated 
into  an  analysis  process)  while  others  viewed  it  as  a  negative  indicator  (e.g.,  as  indicating 
that  the  analyst  did  not  have  a  strong  command  of  the  subject  matter  and  thus  needed  to 
seek  out  expert  support).  This  contrasting  interpretation  of  similar  cues  was  found  in 
participant's  comparisons  of  briefing  reports  as  well  as  comparisons  of  process  documents. 

This  finding  echoes  an  observation  shared  by  a  number  of  participants  in  the  study — that 
there  is  no  one  right  way  to  do  analysis.  Rather,  an  analyst  must  accept  that  others  have 
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different  practices  that  produce  acceptable  results  and  that  their  approaches  can  be  equally 
valid,  even  if  not  completely  understood.  As  described  by  one  study  participant: 

"I'm  not  sure  that  there  is  one  correct  method  to  doing  analysis.  Sometimes,  it's 
understanding,  or  not  even  understanding,  accepting  that  someone  else  has 
practices  that  are  different  than  yours.  But  when  they  follow  those  practices,  they 
come  out  with  solid  results:  that  they're  going  to  be  repeatable,  that  they're  going 
to  be  verified...  that  they're  going  to  be  valid.  And  it's  different  from  mine;  and  I 
may  disagree  with  the  way  they  got  it  or  may  not  understand  how  they  did  it,  but 
they  still  come  out  with  valid  results. " 

This  perspective  seemed  to  influence  how  participants  attended  to  briefing  report  cues  as 
well  as  in  how  they  attended  to  process  document  cues.  In  relation  to  comparison  of 
reports,  participants  attended  both  to  product  formatting  details,  such  as  writing  style,  and 
to  more  subtle  nuances  derived  from  the  content  of  the  reports,  such  as  the  inferences  that 
could  be  made  about  the  perspectives  and  biases  of  author. 

In  comparing  process  documents,  participants  generally  preferred  to  use  "Key  Documents" 
and  "Hypotheses  Considered"  in  assessing  analytic  rigor.  It  is  perhaps  not  surprising  that 
primary  sources  and  hypotheses  were  viewed  as  being  most  critical  in  assessing  rigor.  From 
the  perspective  of  participants,  checking  key  documents  was  much  the  same  as  checking 
sources — which  is  among  the  most  basic  approaches  a  professional  analyst  is  trained  to  use. 
The  importance  of  hypotheses,  then,  are  also  unsurprising  when  viewed  in  light  of  the 
perspective  among  some  in  the  intelligence  community  that  the  analysis  of  hypotheses  is 
the  primary  path  to  analytical  rigor  (Heuer,  1999).  In  fact,  this  finding  lends  validity  to  this 
perspective,  as  it  seems  some  professional  analysts  do  perceive  an  understanding  of 
hypotheses  as  being  an  important  component  of  judging  rigor. 

Relative  to  the  comparison  of  process  documents  prompt,  there  is  also  something  to  be 
learned  from  what  participants  did  not  prefer.  In  the  study,  analysts  generally  found  the 
"Where  Documents  Stored  in  Folders"  process  documents  to  have  very  little  value  in 
assessing  rigor.  What  became  apparent  in  the  discussions  about  this  type  of  document,  was 
that  the  participants  were  generally  not  interested  in  how  analysts  carried  out  their  analysis 
processes,  but  rather,  were  interested  in  understanding  why  analysts  did  what  they  did  in 
their  analysis.  One  participant,  for  example,  commented  that,  while  seeing  the  hypotheses 
considered  was  useful,  it  would  have  been  more  valuable  to  see  how  those  hypotheses 
changed  during  the  course  of  the  analysis  process — in  the  form  a  "hypothesis  timeline"  for 
example. 

There  are  two  general  findings  that  emerged  from  this  study  regarding  the  use  of  cues  in 
assessing  analytic  rigor.  First,  participants  tended  to  use  the  same  cues  in  judging  rigor,  but 
interpreted  them  in  diverse  ways.  Second,  in  judging  rigor,  participants  were  more 
interested  in  process  information  cues  that  could  address  questions  of  "why"  rather  than 
those  that  could  address  questions  of  "what." 

The  finding  that  key  documents  and  hypotheses  were  deemed  most  critical  in  judging  rigor 
is  somewhat  limited  in  generalizability,  however,  as  it  is  unclear  whether  these  cues  were 
identified  as  being  important  because  of  factors  acting  within  the  intelligence  community 
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or  whether  they  reflect  critical  cues  in  judging  rigor  across  information  analysis  domains. 
While  this  research  does  not  purport  to  have  identified  the  cues  that  contribute  to 
assessments  of  rigor,  it  has  helped  to  refine  the  general  understanding  of  these  cues,  by 
exploring  the  aspects  of  analysis  products  and  processes  that  are  meaningful  to  professional 
analysts  in  judging  analytic  rigor. 

4.2.3  Informing  the  Design  of  Analysis  Support  Tools 

Given  the  inherent  challenges  of  building  and  evaluating  intelligence  analysis  support  tools, 
an  understanding  of  rigor  is  valuable  in  informing  the  design  of  software  to  assist  the 
professional  analyst  (Elm  et  al.,  2005;  Greitzer,  2005;  Billman,  Convertino,  Shrager,  Massar,  & 
Pirolli,  2006;  Morse,  Steves,  &  Scholtz,  2006).  The  findings  of  this  research  aid  tool 
development  in  two  primary  ways. 

The  first  insight  from  the  study  is  that  developers  should  design  current  support  tools  to 
reveal,  rather  than  mask,  the  rigor  of  the  analysis  process.  The  catch,  however,  is  that 
conveying  an  understanding  of  process  is  not  the  same  as  simply  providing  access  to 
information  about  the  process,  in  as  much  as  providing  observability  is  not  the  same  as 
merely  making  data  available  (Woods,  Patterson,  &  Roth,  2002).  In  fact,  providing  decision 
makers  with  ambiguous  information  about  analytic  process  is  apt  to  have  the  opposite 
impact  of  that  desired,  serving  only  to  reinforce  assessments  of  perceived  rigor  (Lord,  Ross, 

&  Lepper,  1979).  Thus,  process  information  must  be  framed  in  such  a  way  that  it  conveys 
process  insight,  revealing  the  effective  rigor  of  an  analysis  process  in  a  way  that  challenges 
conceptions  of  perceived  rigor. 

The  second  way  this  research  supports  the  design  process  is  through  the  finding  that  process 
insight  does,  in  fact,  have  an  influence  on  judging  rigor.  Thus,  this  research  potentially 
inspires  the  design  of  a  totally  new  class  of  innovative  tools  that  afford  capturing  and 
sharing  of  the  analytic  rigor  inherent  in  intelligence  analysis  processes.  In  either  case, 
additional  research  would  be  required  in  order  to  better  define  the  critical  components  of 
rigor  that  need  to  be  captured  in  building  an  effective  analysis  support  tool.  However,  the 
findings  of  this  study  also  provide  initial  guidance  for  such  efforts,  offering  some  initial 
insights  into  the  nature  of  the  cues  that  are  used  to  infer  analytic  rigor. 

4.2.4  Extending  the  Findings  Across  Information  Analysis  Domains 

Perhaps  the  most  surprising  finding  from  the  study  was  that — despite  significant  efforts  in 
developing  the  scenario-based  analyses  and  despite  inputs  from  professional  analysts  in 
refining  them  as  plausible  analytic  responses — very  few  of  the  participants  thought  either  of 
the  analysis  reports  was  ready  to  send  forward.  As  described  in  the  previous  section,  before 
reviewing  the  process  documents  only  three  participants  thought  that  either  report  might 
be  ready  to  send  forward.  After  seeing  the  process  documents  that  number  decreased,  as 
only  one  participant  thought  either  of  the  reports  might  be  ready  to  go  forward.  More 
surprising  still  was  the  fact  that  this  participant  perceived  the  A(2)  report — generated  by  the 
low  rigor  analysis  process — as  being  more  ready  to  send  forward  than  the  A(1)  report. 

The  consequence  of  this  finding,  then,  was  a  reconceptualization  of  the  definition  of  rigor 
in  intelligence  analysis  that  contrasts  with  the  traditional  perspective.  Rather  than  focusing 
on  deviation  from  a  baseline  process,  study  participants  instead  identified  a  number  of 
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different  attributes  that  compose  a  rigorous  analysis.  Thus,  rigor  was  not  viewed  as  a  single 
variable  that  reflected  adherence  to  a  particular  process,  but  rather  it  was  a  composite, 
emergent  property  of  many  different  aspects  of  process.  The  common  theme  across  these 
aspects  of  the  analytic  process  was  that  rigor  was  consistently  framed  as  a  measure  of 
sufficiency.  Thus,  one  of  the  more  powerful  implications  of  this  work  arises  from  the 
recognition  that  supporting  an  understanding  of  rigor  is  about  supporting  an 
understanding  of  the  sufficiency  of  analysis  in  context,  rather  than  about  determining 
adherence  to  standard  process. 

This  finding  on  rigor  is  meaningfully  applied  beyond  the  intelligence  analysis  community,  as 
the  important  role  of  rigor  is  acknowledged  in  other  information  analysis  contexts  as  well. 

In  perhaps  the  most  salient  example,  described  in  Section  1,  the  importance  of  effectively 
communicating  analytical  rigor  was  brought  to  the  forefront  of  NASA's  focus  following  the 
Columbia  accident  (Columbia  Accident  Investigation  Board,  2003;  Crippen,  et.  al.,  2005; 
Woods,  2005).  Private  industry  too  is  increasingly  recognizing  the  value  of  understanding 
the  appropriate  level  of  rigor  in  information  analysis  processes,  relative  to  decision  criticality 
and  resource  priority  (Krizan,  1999;  Dube  &  Pare,  2003).  In  health  care,  still  others  question 
the  value  of  the  traditional  understanding  of  rigor  in  relation  to  qualitative  social  research 
(Davies  &  Dodd,  2002).  Thus,  the  importance  of  recognizing  rigor  as  a  sufficiency  metric, 
rather  than  process  deviation  metric,  is  a  critical  finding  that  extends  meaningfully  across 
information  analysis  domains. 


4.3  Rigor  in  Intelligence  Analysis 

As  noted  in  the  previous  section,  the  surprising  finding  that  few  participants  judged  either 
of  the  scenario  reports  as  being  ready  to  forward  to  a  decision  maker  led  to  a  reframing  of 
the  concept  of  rigor  as  a  measure  of  sufficiency,  rather  than  deviation.  This  finding  also 
revealed  that  participants  viewed  rigor  as  an  aggregate  metric  that  emerged  from  the 
interaction  of  multiple  facets  of  an  analytic  process.  This  feedback,  in  turn,  pointed  toward 
a  very  different,  composite  model  for  understanding  rigor  as  a  valuable  concept  in 
supporting  intelligence  analysis.  In  proposing  this  revised  model  of  rigor,  it  is  contrasted 
with  the  traditional  understanding  of  rigor  in  intelligence  analysis. 

As  noted  in  Section  1,  defining  rigor  is  difficult  in  most  any  information  analysis  setting. 
Intelligence  analysis  proves  no  exception,  as  the  diverse  nature  of  analysis  work  makes  the 
identification  of  a  standard  process  that  can  be  applied  across  all  intelligence  contexts 
inherently  difficult  (Berkowitz  &  Goodman,  1991).  In  tactical  military  intelligence 
operations.  Intelligence  Preparation  of  the  Battlefield  (IPB)  is  often  cited  as  the  current 
standard  for  supporting  the  Military  Decision  Making  Process  (MDMP)  (Intelligence 
Preparation  of  the  Battlefield,  1994;  Medby  &  Glenn,  2002).  In  strategic  analysis — where, 
arguably,  a  similarly  well-defined  intelligence  process  does  not  exist — the  identification  of  a 
comparable  standard  is  ambiguous.  There  is,  however,  a  perspective  among  many  in  the 
intelligence  community  that  achieving  rigor  in  the  analysis  process  depends,  most  centrally, 
on  the  application  of  a  formal  method  for  weighing  and  comparing  of  hypotheses — 
commonly  termed  the  Analysis  of  Competing  Hypotheses  (ACH). 
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Heuer  (1999)  defined  the  ACH  approach  as  an  eight  step  method  that  is  "a  rational, 
systematic  process  that  avoids  some  common  analytical  pitfalls"  and  that  is  designed  to 
"guarantee  an  appropriate  process  of  analysis."  Schum  (1987)  also  recognized  the  important 
role  of  hypotheses,  identifying  them  as  one  of  the  three  "necessary  ingredients  of  inferential 
intelligence  analysis,"  in  conjunction  with  evidence  and  assumptions.  Although,  Schum  (1987) 
tended  to  emphasize  how  different  types  of  evidence  support  or  contradict  individual 
hypotheses,  common  across  both  approaches  is  the  understanding  that  rigorous  analytic 
process  is  attained  through  a  structured,  deliberate  connection  of  data  to  hypotheses  and 
the  subsequent  comparison  among  viable  alternatives. 

Generally,  advocates  of  this  perspective  support  the  more  widespread  application  of 
structured  methodologies,  such  as  ACH,  contending  that  their  use  is  required  to  achieve 
rigor  in  the  analytic  process  (Pirolli,  Good,  Heiser,  Sharger,  &  Hutchins,  2006).  Folker  (2000), 
for  example,  conducted  an  exploratory  study  with  analysts  from  four  different  joint 
intelligence  centers,  finding  that  analysts  who  employed  even  a  simplified  hypothesis 
testing  method  outperformed  those  who  did  not. 

There  is,  however,  also  the  perspective  among  many  in  the  intelligence  community  that  the 
application  of  the  ACH  method  is  impractical,  among  other  criticisms,  and  thus  not  a 
reasonable  standard  for  assessing  rigor  (Czerwinski,  1998).  Practitioners  holding  this 
perspective  contend  that — as  observed  by  Johnson  (2005) — the  use  of  structured 
methodologies  are  too  restrictive  to  be  routinely  implemented  in  the  "tradecraft"  of 
intelligence  analysis,  noting  studies  that  have  found  little,  if  any,  benefit  to  using  such 
methods  (e.g.,  Cheikes,  Brown,  Lehner,  &  Alderman,  2004). 

Even  the  dichotomous  views  on  rigor  presented  here  are  inherently  incomplete,  however,  as 
there  are  still  others  in  the  intelligence  community  who  have  entirely  different  perspectives 
on  analytic  rigor  (e.g.,  Moore  &  Krizan,  2001).  The  important  overall  message  then,  is  that 
the  concept  of  rigor  in  intelligence  analysis — as  in  many  other  information  analysis  domains 
— is  traditionally,  albeit  debatably,  viewed  as  adherence  to  a  formalized  standard  process. 
The  results  of  the  study,  however,  support  a  different  perspective  on  rigor.  None  of  the 
participants  in  the  study,  in  fact,  directly  questioned  whether  or  not  the  junior  analysts  in 
the  scenario  followed  a  specific  method,  such  as  ACH,  in  assessing  the  analyses.  Instead,  the 
participants  sought  to  understand  many  different,  but  interrelated  aspects  of  their  analysis 
processes  in  judging  rigor. 

In  contrast  with  the  traditional  perspective,  the  revised  definition  of  rigor  that  emerged 
from  the  study  frames  the  concept  as  the  composite  of  multiple  process  dimensions.  This 
multi-attribute  metric  characterizes  these  indicators  as  independent  components  of  the 
analysis  process  which,  when  aggregated,  reveal  a  composite  assessment  of  analytic  rigor.  In 
addition  to  the  findings  from  the  study,  this  framing  of  rigor  was  shaped  by  the  traditional 
perspectives  on  rigor  in  intelligence  analysis,  the  findings  from  other  studies  of  professional 
intelligence  analysts,  feedback  from  a  diverse  group  of  professional  analysts,  and  empirical 
insights  that  emerged  from  interactions  with  those  analysts,  all  of  which  occurred  during 
the  study  or  as  follow-ups  to  the  study. 
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Attribute 


Description 


A(1)  A(2) 


Hypothesis  Hypothesis  Exploration  describes  the  extent  to  which  multiple  L  L 

Exploration  hypotheses  were  considered  in  explaining  data.  In  a  low-rigor 
process  there  is  minimal  weighing  of  alternatives.  A  high-rigor 
process,  in  contrast,  involves  broadening  of  the  hypothesis  set 
beyond  an  initial  framing  and  incorporating  multiple  perspectives  to 
identify  the  best,  most  probable  explanations. 

Information  Information  Search  relates  to  the  depth  and  breadth  of  the  search  H  M 

Search  process  used  in  collecting  data.  A  low-rigor  analysis  process  does  not 

go  beyond  routine  and  readily  available  data  sources,  whereas  a 
high-rigor  process  attempts  to  exhaustively  explore  all  data 
potentially  available  in  the  relevant  sample  space. 

Information  Information  Validation  details  the  level  at  which  information  sources  M  M 

Validation  are  corroborated  and  cross-validated.  In  a  low-rigor  process  little 

effort  is  made  to  use  converging  evidence  to  verify  source  accuracy, 
while  a  high-rigor  process  includes  a  systematic  approach  for 
verifying  information  and,  when  possible,  ensures  the  use  of  sources 
closest  to  the  areas  of  interest. 

Stance  Stance  Analysis  is  the  evaluation  of  data  with  the  goal  of  identifying  M  L 

Analysis  the  stance  or  perspective  of  the  source  and  placing  it  into  a  broader 

context  of  understanding.  At  the  low-rigor  level  an  analyst  may 
notice  a  clear  bias  in  a  source,  while  a  high-rigor  process  involves 
research  into  source  backgrounds  with  the  intent  of  gaining  a  more 
subtle  understanding  of  how  their  perspective  might  influence  their 
stance  toward  analysis-relevant  issues. 

Sensitivity  Sensitivity  Analysis  considers  the  extent  to  which  the  analyst  M  M 

Analysis  considers  and  understands  the  assumptions  and  limitations  of  their 

analysis.  In  a  low-rigor  process,  explanations  seem  appropriate  and 
valid  on  a  surface  level.  In  a  high-rigor  process  the  analyst  employs  a 
strategy  to  consider  the  strength  of  explanations  if  individual 
supporting  sources  were  to  prove  invalid. 

Specialist  Specialist  Collaboration  describes  the  degree  to  which  an  analyst  M  L 

Collaboration  incorporates  the  perspectives  of  domain  experts  into  their 

assessments.  In  a  low-rigor  process  little  effort  is  made  to  seek  out 
such  expertise,  while  in  a  high-rigor  process  the  analyst  has  talked  to, 
or  may  be,  a  leading  expert  in  the  key  content  areas  of  the  analysis. 

Information  Information  Synthesis  refers  to  how  far  beyond  simply  collecting  and  M  H 

Synthesis  listing  data  an  analyst  went  in  their  process.  In  the  low-rigor  process 

an  analyst  simply  complies  the  relevant  information  in  a  unified 
form,  whereas  a  high-rigor  process  has  extracted  and  integrated 
information  with  a  thorough  consideration  of  diverse  interpretations 
of  relevant  data. 

Explanation  Explanation  Critique  is  a  different  form  of  collaboration  that  M  L 

Critique  captures  how  many  different  perspectives  were  incorporated  in 

examining  the  primary  hypotheses.  In  a  low-rigor  process,  there  is 
little  use  of  other  analysts  to  give  input  on  explanations  quality.  In  a 
high-rigor  process  peers  and  experts  have  examined  the  chain  of 
reasoning  and  explicitly  identified  which  inferences  stronger  and 
which  are  weaker. 

Note.  "H"  reflects  high  rigor,  "M"  reflects  moderate  rigor,  and  "L"  reflects  low  rigor  in  an  attribute  of  the 
process  for  Analysis  Process  1  [A(1 )]  and  Analysis  Process  2  [A(2)]  developed  for  the  LNG  Scenario. 


Table  1 5.  Dimensions  of  Rigor  as  Applied  to  the  Study  Analysis  Processes 
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Specifically,  the  findings  of  the  study  led  to  the  identification  of  eight  critical  attributes  of 
analysis  processes  that  contribute  to  assessments  of  rigor,  as  shown  in  Table  1 5.  Also 
included  in  the  table  are  assessments  of  the  analyses  developed  for  the  LNG  Scenario,  relative 
to  the  proposed  critical  aspects  of  rigor,  that  are  based  on  feedback  generated  by  study 
participants.  Interestingly,  this  reassessment  of  the  scenario-based  analyses  helps  explain 
why  there  was  disagreement  among  study  participants  as  to  which  analysis  was  most 
rigorous — and  why  so  few  were  reluctant  to  pass  them  on  in  the  Supervisor's  Dilemma. 

While  A(1)  and  A(2)  were  designed  to  reflect  high  and  low  rigor  processes  respectively, 
within  the  revised  framing  the  two  processes  actually  differed  more  in  their  patterns  of 
rigor  than  in  absolute  levels  of  rigor.  Despite  collaborative  design  efforts,  A(1),  it  turned  out, 
did  not  contain  high  levels  of  rigor  across  all  dimensions,  but  rather,  showed  generally 
moderate  rigor  in  most  aspects  of  the  process,  while  in  fact  even  being  low  in  one  aspect.  A 
(2)  turned  out  to  be  low  in  many  dimensions,  as  was  intended,  but  also  showed  moderate  or 
high  rigor  in  a  few  dimensions. 

This  finding,  perhaps,  explains  why  some  participants  perceived  the  A(2)  report  as  being  the 
better  report,  even  after  seeing  process  information — they  attended  to  and  valued  certain 
aspects  of  analytic  rigor  over  others.  One  possible  interpretation  of  this  finding  is  that,  while 
it  is  relatively  easy  to  reach  a  moderate  level  of  analytical  rigor  in  information  analysis,  it  is 
often  quite  hard  to  reach  a  high  level  of  rigor  in  more  than  a  couple  dimensions.  The 
corollary  to  this  interpretation  is  that  it  is  also  quite  easy  to  overlook  an  aspect  of  analysis, 
remaining  low  on  a  dimension  of  an  otherwise  moderate  level  analysis. 

The  general  message,  then,  is  that  rigor  in  information  analysis  is  not  understood  by 
assessing  deviations  of  process,  but  rather,  by  assessing  the  contextual  sufficiency  of  many 
different  aspects  of  process.  The  results  of  the  study  were  further  used  to  identify  these 
critical  aspects  of  the  analysis  process,  leading  to  a  revised  definition  of  rigor  that 
encompasses  the  eight  dimensions  of  process  listed  in  Table  1 5.  This  proposed  definition  of 
rigor  implies  that,  to  the  degree  that  there  is  rigor  in  an  analysis,  an  individual  assessing  an 
analysis  process  would  want  to  know  something  about  each  of  these  attributes  in  forming  a 
composite  judgment  of  the  sufficiency  of  rigor.  However,  in  positing  this  definition  of  rigor, 
it  is  noted  that,  as  discussed  in  the  next  section,  the  generalizability  of  this  and  other 
findings  of  the  study  may,  in  fact,  be  somewhat  constrained  due  to  both  pragmatic  and 
methodological  limitations  of  study  design. 


4.4  Limitations  of  Study  Design 

The  research  study  discussed  in  this  thesis  was  designed  to  explore  the  understanding  of 
rigor  in  information  analysis  in  the  applied  setting  of  professional  intelligence  analysis. 
While  there  are  many  reasons,  as  discussed  in  Section  2,  that  the  domain  of  intelligence 
analysis  is  a  representative  context  for  the  study  of  information  analysis,  there  are,  of 
course,  limitations  in  the  study  design  that  restrict  the  generalizability  of  findings  both 
within  the  domain  of  intelligence  analysis  and  across  domains  of  information  analysis. 

Many  of  the  limitations  resulted  from  pragmatic  tradeoffs  made  in  the  course  of  designing 
and  running  the  study.  For  example,  as  described  earlier,  the  LNG  Scenario  was  selected 
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based  on  its  authenticity  as  a  generic  information  analysis  task,  rather  than  its  validity  as  a 
realistic  intelligence  analysis  task.  This  reflected  a  decision  to  favor  obtaining  candid 
feedback  over  representing  true  domain  authenticity,  as  selecting  a  scenario  topic  that  was 
of  real  professional  interest  to  a  participant  analyst  would  have,  by  definition,  sharply 
limited  his  or  her  ability  to  discuss  it  in  the  unclassified  setting  of  the  study.  This  tradeoff,  in 
conjunction  with  time  constraints,  also  influenced  the  scope  of  the  study,  as  the  analysis 
processes  and  briefing  reports  critiqued  by  participants  were  smaller  in  scale  than  those  that 
analysts  actually  deal  with  on  a  day-to-day  basis. 

Another  pragmatically-driven  tradeoff  was  in  defining  the  population  sample  size  of  the 
study.  Sample  size  was  limited  most  notably  by  the  inherent  difficulties  of  identifying 
representative  analysts  to  participate  in  the  research  study.  Not  only  do  professional 
strategic  intelligence  analyst  represent  a  very  small  population,  but  the  nature  of  their  work 
is  such  that  they  are  necessarily  restricted  from  talking  much  about  it.  Additionally,  given 
the  immense  importance  and  continuous  pace  of  intelligence  analysis,  most  analysts  have 
limited  time  available  to  commit  to  research  purposes.  Consequently,  it  should  be  noted 
that  any  research  within  the  domain  of  intelligence  analysis  will  be  limited  in  its  ability  to 
secure  a  representative  sample  of  participants.  And  thus,  the  nature  of  the  study  sample 
might  also  be  viewed  a  strength,  in  that  it  was  possible  to  identify  a  pool  of  highly 
experienced  analysts  to  participate  in  an  exploratory  research  study. 

Other  tradeoffs  relating  to  study  design  were  more  methodological  in  nature.  One 
limitation  of  note  was  that,  given  that  there  was  no  prescreening  of  participants,  it  was 
difficult  to  determine  the  extent  to  which  perceptions  about  the  LNG  Scenario  may  have 
biased  judgments  of  rigor.  While  this  risk  was  somewhat  reduced  by  the  fact  that  no 
participant  indicated  a  professional  familiarity  with  the  subject  matter,  there  are  certainly 
related  issues  that  participants  may  have  felt  strongly  about  that  could  have  predisposed 
them  toward  favoring  one  report  over  the  other — for  example,  their  stance  toward 
dependence  on  foreign  fuels  or  the  impact  of  industrialization  on  the  environment. 

The  potential  influence  of  predisposition  and  bias  on  perceptions  of  analytical  rigor  is 
certainly  important,  although  not  new  or  unique  to  this  study.  Research  in  social  psychology 
has  long  observed  the  tendency  for  people  to  perceive  as  more  credible  and  valid  those 
opinions  that  support  their  prior  beliefs  on  an  issue  (Lord,  Ross,  &  Lepper,  1979;  Pyszczynski, 
Greenberg,  &  Holt,  1985).  This  tendency  relates  more  broadly  to  the  second  of  Kahn's  (2001) 
two  "all-encompassing,  never  ending...  [and]  ultimately  unresolvable"  problems  in  intelligence: 
"how  to  get  statesmen  and  generals  to  accept  information  that  they  do  not  like,"  (the  first 
problem  being  the  inherent  difficulties  in  making  predictions  about  the  future  in  an 
uncertain,  dynamic  world).  Unfortunately,  this  is  not  a  problem  with  much  promise  of  ever 
being  fully  resolved,  although  there  is  some  cause  to  believe  that  the  negative  impact  of 
such  a  tendency  might  be  assuaged  through  a  better  understanding  of  rigor.  In  fact,  it  is 
conceivable  that  future  studies  of  rigor  in  information  analysis  would  explore  the  influence 
of  this  very  factor. 

A  second  methodological  tradeoff,  the  scenario-embedded  Elicitation  by  Critique  approach, 
as  a  form  of  Staged  World  study  (Woods  &  Hollnagel,  2006),  was  used  because  it  balanced 
the  inability  to  directly  observe  (or  talk  about)  real  analysts  doing  real  analysis  work  with 
the  desire  to  explore  the  role  of  rigor,  rather  than  test  a  specific  hypothesis  about  rigor. 
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Additionally,  the  EBC  method  used  in  the  study  was  modified  in  order  to  incorporate  the 
LNG  Scenario,  another  manifestation  of  the  previously  mentioned  tradeoff  between  ability 
to  collect  data  and  task  authenticity.  Of  course,  there  are  inherent  tradeoffs  regarding 
generalizability  of  findings  in  choosing  any  study  methodology  over  any  other.  In  selecting 
the  design  of  this  study,  it  was  determined  that  the  theoretical  tradeoffs  stemming  from  the 
method  were  appropriate  relative  to  the  broader  goals  of  the  research. 

There  are  still  other  study  limitations  that  resulted  from  both  pragmatic  and  methodological 
considerations,  which  perhaps  more  seriously  restrict  the  ability  to  generalize  the  findings. 
Most  notably,  while  one  of  the  stated  goals  of  the  study  was  to  better  understanding  the 
role  of  rigor  in  enhancing  the  connection  between  analysts  and  decision  makers,  it  should 
be  noted  that  no  decision  makers  participated  in  the  study.  From  a  practical  standpoint,  it 
would  have  arguably  been  even  more  difficult  to  identify  truly  representative  decision 
makers  to  participate  in  this  study,  as  they  represent  an  even  smaller  and  more  time 
constrained  population  than  professional  analysts.  From  a  theoretical  standpoint,  an 
assumption  was  made  that  the  way  in  which  analysts  and  decision  makers  judge  rigor  was 
similar,  at  least  in  terms  of  the  most  critical  dimensions. 

It  is  a  very  valid  and  significant  criticism  to  question  the  appropriateness  of  this  assumption 
that  analysts  and  decision  makers  view  rigor  similarly.  In  the  context  of  the  study,  the  impact 
of  this  assumption  was  somewhat  mitigated  by  asking  participants  to  critique  the  analysis 
reports  from  the  assumed  role  of  supervisor.  In  intelligence  analysis,  the  analyst  supervisor  in 
many  instances  serves  as  intermediary  between  analysts  and  decisions  makers  and  thus  must 
view  briefing  reports  with  both  perspectives  in  mind.  Of  course,  one  might  argue  that  your 
average  analyst  is  unable  to  truly  assume  the  perspective  of  a  supervisor — a  position  mildly 
supported  by  the  results  of  the  study. 

Incidentally,  a  couple  study  participants,  although  possessing  significant  prior  experience  in 
the  role  of  analyst,  were  in  their  present  duties  at  the  time  of  the  study  considered  to  be 
supervisors.  Interestingly,  these  participants  were,  at  least  initially,  more  likely  to  identify 
the  A(2)  briefing  as  being  the  better,  more  rigorous  report.  Given  that  the  A(2)  report  was 
generally  judged  by  participants  as  being  better  relative  to  the  overall  presentation 
qualities,  it  is  tempting  to  conclude  that  there  was,  in  fact,  a  difference  in  perspective 
between  analysts  and  supervisors.  The  veracity  of  this  conclusion  is  tempered,  however,  by 
the  fact  that  there  were  only  two  known  analyst  supervisors  who  participated  in  the  study, 
making  it  difficult  to  determine  whether  this  difference  was  significant  or  not. 

Still,  it  could  be  reasonably  concluded  that  this  finding  at  least  lends  credence  to  the 
position  that  analysts  and  decision  makers  judge  rigor  in  very  different  ways.  Of  course,  it 
could  also  be  reasonably  concluded  that,  in  a  broad  sense,  the  assumption  is  not  all  that 
consequential,  given  that  it  is  still  valuable  to  understand  how  analysts  assess  rigor,  even  if 
only  to  contrast  their  perspective  with  future  research  exploring  how  "real"  supervisors  and 
decision  makers  assess  rigor.  Although  unresolved,  this  debate  is  perhaps  even  more 
relevant  when  viewed  in  light  of  an  alternative  perspective  on  the  role  of  rigor  within  the 
intelligence  community,  as  discussed  in  the  following  section. 
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4.5 


An  Alternative  Perspective  on  Rigor 


There  is  another  perspective  in  the  intelligence  community  on  the  value  of  understanding 
rigor  that  warrants  mention.  Simply  put,  this  alternative  perspective  asserts  that  an 
understanding  of  rigor  is  not  what  matters — particularly  to  decision  makers — and  thus  is  not 
a  valuable  aspect  of  the  intelligence  cycle  to  explore.  As  framed  by  Betts  (1978),  the 
problem  is  this: 

Intelligence  consumers  are  political  men  who  have  risen  by  being  more  decisive  than 
reflective,  more  aggressive  than  introspective,  and  confident  as  much  as  cautious... 

Even  if  they  could  be  forced  to  confront  scholarly  evidence  of  the  dynamics  of 
misperception,  it  is  uncertain  that  they  could  consistently  internalize  it...  Moreover, 
the  line  between  perception  and  judgement  is  very  thin,  and  consumers  cannot 
carefully  scrutinize,  compare,  and  evaluate  the  methodologies  of  competing 
analyses,  for  the  same  prosaic  reasons  (the  problem  of  expertise  aside)  that  impedes 
many  proposed  reforms:  they  do  not  have  the  time  to  do  so.  Solutions  that  require 
principals  to  invest  more  attention  than  they  already  do  are  conceptually  valid  but 
operationally  weak.  Ideally,  perhaps,  each  principal  should  have  a  Special  Assistant 
for  Rigor  Enforcement. 

In  fact,  some  participants  in  the  study  echoed  this  very  sentiment.  In  its  mildest  form,  some 
indicated  that  a  decision  maker  simply  would  not  look  for  indicators  of  rigor.  Other 
participants  expressed  a  variant  on  this  view,  noting  that  process  quality  does  not  change 
whether  it  is  a  good  or  bad  product.  As  a  more  extreme  position,  one  participant 
commented  that  often  times,  given  the  high  visibility  of  their  position,  policy  makers  do  not 
care  much  if  the  information  provided  in  an  analysis  is  "right,"  as  it  gives  them  somebody  to 
blame  if  things  go  wrong. 

In  addressing  this  alternative  perspective  on  rigor,  there  are  at  least  two  ways  that  one 
might  respond.  The  first  way  would  be  to  assert  that  rigor  is  still  a  valuable  concept  in  the 
domain  of  intelligence  analysis — just  not  to  decision  makers.  Instead  of  being  applied  to 
enhance  the  connection  between  analysts  and  decision  makers,  an  understanding  of  rigor 
would  perhaps  more  usefully  be  applied  in  supporting  the  connection  between  analysts  and 
supervisors  or  in  analyst-to-analyst  interactions. 

As  discussed  previously,  the  Supervisor's  Dilemma  was,  to  a  certain  extent,  developed  to 
address  this  very  issue.  By  framing  the  dilemma  as  belonging  wholly  to  the  supervisor,  a 
tradeoff  was  made  in  selecting  a  spot  in  the  information  analysis  and  planning  cycle  that,  in 
actuality,  is  sometimes  real  and  sometimes  notional.  Thus,  it  does  not  really  matter  exactly 
where  in  the  cycle  the  decision  is  made  to  invest  resources  or  not,  the  more  critical  point  is 
that  somewhere  in  the  chain  this  decision  is  being  made,  whether  explicitly — as  in  the  study 
— or  implicitly — as  in  the  case  of  the  analyst  who  judges  that  an  analysis  is  sufficient  relative 
to  resource  pressures  and  time  limitations. 

A  second  way  to  respond  to  this  alternative  perspective  would  simply  be  to  restate  that,  as 
supported  by  the  findings  of  this  study,  an  understanding  of  rigor  is  in  fact  valuable  in 
influencing  perceptions  of  rigor.  As  noted  in  the  quote  above,  Betts  concedes  that  such  an 
approach  is  likely  to  be  conceptually  valid  but,  by  creating  additional  demands  for  decision 
makers,  it  is  also  likely  to  err  in  its  implementation.  Consequently,  the  underlying  issue  may 
not  be  whether  or  not  rigor  is  a  valuable  concept,  but  whether  or  not  the  concept  can  be 
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incorporated  in  a  way  that  it  makes  judging  the  sufficiency  of  an  analytic  process  easier  and 
more  intuitive,  rather  than  more  complicated  and  nuanced. 

Thus,  the  general  response  to  the  alternative  perspective  on  rigor  is  simple:  rigor  is 
important  because  it  shows  the  potential  to  be  useful  as  a  general  concept.  However,  in 
positing  this  view,  it  is  also  acknowledged  that  an  understanding  of  rigor  is  not  all  that 
matters  in  improving  the  broader  intelligence  cycle,  but  one  of  many  things  that  matter — a 
perspective  that  is  apt  to  hold  in  other  information  analysis  domains  as  well.  The  Doctrine  for 
Intelligence  Support  to  Joint  Operations  (2000),  for  example,  identifies  other  critical  attributes 
of  intelligence — such  as  timeliness,  usability,  and  relevance — that  extend  beyond  the  basic 
framing  of  analytical  rigor  advanced  in  this  thesis.  Thus,  as  it  is  very  much  a  part  of  what  is 
important,  the  understanding  of  rigor  is  a  concept  warranting  continued  exploration. 


4.6  Future  Work 

As  much  as  the  findings  of  this  research  serve  to  refine  the  understanding  of  rigor  in 
information  analysis,  so  too  do  they  expand  the  potential  directions  of  future  explorations 
of  the  concept.  The  study  prompted  three  main  directions  for  future  inquiry.  First,  within 
the  context  of  intelligence  analysis,  the  research  findings  open  the  door  for  future 
explorations  of  the  process  cues  that  are  used  in  judging  rigor.  Such  studies  would  focus 
more  specifically  on  understanding  which  cues  are  most  relevant  in  judging  rigor  and  on 
understanding  how  these  cues  could  be  leveraged  to  support  the  professional  intelligence 
analyst.  Additionally,  future  research  might  also  involve  a  study  of  rigor  using  decision 
makers,  rather  than  analysts,  looking  at  how  their  perspectives  are  similar  and  different  in 
forming  judgments  of  rigor. 

Second,  because  the  research  was  based  only  on  findings  developed  in  the  context  of 
intelligence  analysis,  future  work  would  focus  on  understanding  how  the  findings  of  this 
study  translate  into  other  information  analysis  domains.  Such  research  may  run  similar 
studies  to  that  described  in  this  thesis  or  may  include  entirely  different  studies  that  explore 
other  aspects  of  rigor  in  other  information  analysis  contexts.  In  either  case,  the 
methodological  contributions  of  this  study  could  prove  invaluable,  as  the  EBC  methodology 
that  was  refined  around  the  Supervisor's  Dilemma  is  well  suited  for  application  in  other 
information  analysis  domains,  while  the  LNG  Scenario  offers  a  developed  cognitive  case  that 
is  flexible  enough  to  be  adapted  for  other  research  efforts. 

Finally,  the  revised  definition  of  rigor  as  an  emergent  and  multi-attribute  property  of 
analysis  serves  as  the  basis  for  the  future  development  of  new  tools  and  techniques  for 
improving  information  analysis.  As  such,  the  potential  directions  for  the  continued  study  of 
rigor  in  information  analysis  are  extensive,  as  the  exploration  of  rigor  is  necessary  to  better 
understanding  how  the  concept  can  be  leveraged  to  improve  analysis  processes  across 
domains  of  information  analysis. 
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4.7 


Conclusions 


As  the  changing  technological  landscape  of  information  analysis  exacerbates  the  already 
difficult  task  of  detecting  when  analysis  is  of  sufficient  rigor,  it  becomes  increasingly  easy  for 
analyst  and  decision  maker  alike  to  be  trapped  by  shallow  analysis — perceiving  an  it  as 
being  deeper  and  more  rigorous  than  it  actually  is.  A  better  understanding  of  rigor  is  an 
analytic  broadening  check  to  be  leveraged  against  this  constant  risk.  The  research  study 
discussed  in  this  thesis  explored  the  concept  of  rigor  with  this  perspective  in  mind, 
producing  many  valuable  contributions — both  in  methodology  and  in  findings — that  serve 
to  advance  the  understanding  of  rigor  in  information  analysis. 

In  the  study,  the  LNG  Scenario  was  developed  as  a  context  for  the  exploration  of  rigor.  As  a 
cognitive  case,  the  scenario  also  shows  the  potential  to  be  leveraged  in  the  exploration  of 
other  aspects  of  information  analysis.  The  Supervisor's  Dilemma  also  emerged  from  the 
development  of  the  study  as  a  valuable  framing  for  investigating  the  link  between  analysis 
and  replanning  in  information  analysis.  The  study  also  served  to  validate  and  refine — by 
embedding  in  the  context  of  a  scenario  walkthrough  study  design — the  Elicitation  by 
Critiquing  methodology  as  a  valuable  approach  for  generating  insights  from  experienced 
practitioners  that  go  beyond  simple  descriptions  of  the  work. 

The  study  also  produced  three  general  findings  that  contributed  to  the  understanding  of 
rigor  in  information  analysis.  First,  the  data  showed  that  providing  process  insight  to 
participants  initiated  change  in  perceptions  of  rigor.  Second,  it  was  found  that  professional 
analysts  tended  to  use  similar  cues  in  making  judgments  of  rigor;  however,  the  ways  in 
which  those  cues  were  transformed  into  composite  assessments  of  rigor  were  more  varied. 
Third,  the  surprising  finding  that  few  study  participants  perceived  either  of  the  scenario- 
based  analysis  reports  as  being  ready  to  forward  to  a  decision  maker  led  to  the 
development  of  a  revised  definition  of  rigor,  reframing  it  as  a  multi-attribute,  emergent 
measure  of  sufficiency  rather  than  as  a  measure  of  process  deviation. 

There  are  many  potential  ways  in  which  these  contributions  to  the  overall  understanding  of 
rigor  might  be  leveraged  to  improve  the  process  of  information  analysis.  Most  notably, 
understanding  rigor  offers  direction  for  avoiding  the  trap  of  shallow  analysis,  by  revealing 
opportunities  to  amplify  the  ability  of  both  analyst  and  decision  maker  to  more 
authentically  judge  the  rigor  of  analysis  products.  Further,  identifying  the  cues  that  are  used 
to  infer  rigor  provides  guidance  for  directly  supporting  the  professional  information  analyst. 
Perhaps  most  importantly  though,  this  research  into  rigor  reveals  promising  directions  for 
the  continued  exploration  of  the  challenges — those  long-standing  as  well  as  those  driven 
by  recent  technological  change — that  confound  the  process  of  information  analysis. 
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************  UNCLASSIFIED  ************** 


As  a  senior  analyst,  you  might  be  asked  to  decide  if  junior  analysts'  reports  are 
ready  to  be  passed  on  to  a  policy-maker.  Please: 

*  Assess  and  compare  the  quality  of  the  two  written  reports 

*  Comment  on  the  “rigor”  of  the  processes  that  produced  each  of  the  reports 

The  two  analysts  were  both  asked:  What  are  the  primary  obstacles  to  using 
Liquified  Natural  Gas  to  address  energy  needs  in  the  U.S.? 
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Figure  A.  1:  Scenario  Introduction  Document,  Page  1  of  1 
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************  UNCLASSIFIED  ************** 


Question:  What  are  the  primary  obstacles  to  using  Liquefied  Natural  Gas  (LNG)  to 
address  energy  needs  in  the  US? 

There  are  varied  uses  for  natural  gas  in  the  United  States.  The  industrial  sector  accounts 
for  the  largest  proportion,  especially  for  power  generation,  but  also  in  the  pulp  and  paper, 
metals,  chemicals,  petroleum  refining,  and  food  processing  industries.  Natural  gas  also 
provides  the  base  ingredients  for  products  such  as  fertilizer,  anti-freeze,  plastic,  and 
fabrics.  In  the  residential  and  commercial  sectors,  natural  gas  is  used  primarily  for 
heating,  cooling,  and  cooking.  The  transportation  sector  is  developing  new  technology 
but  does  not  extensively  use  natural  gas  currently.  Rising  gasoline  prices,  oil  price 
volatility,  and  the  possibility  of  domestic  oil  shortages  have  increased  U.S.  demand  for 
natural  gas,  despite  being  more  expensive  to  produce  and  transport.  Domestic  natural  gas 
reserves  are  no  longer  sufficient  to  satisfy  the  growing  demand,  so  the  U.S.  relies  on 
foreign  natural  gas  imports,  which  tend  to  be  cheaper,  mostly  from  Trinidad. 

Natural  gas  condenses  into  liquid,  or  LNG,  when  it  is  cooled  to  temperatures  below  -260° 
Fahrenheit.  As  a  liquid,  natural  gas  occupies  only  1 /600th  the  volume  of  its  gaseous  state, 
which  allows  it  to  be  stored  and  transported  more  effectively.  LNG  is  then  “regasified” 
when  it  is  wanned.  The  regasification  process  takes  place  at  the  nation’s  four  LNG 
terminals:  Chesapeake  Bay,  Maryland;  Lake  Charles,  Louisiana;  Elba  Island,  Georgia; 
and  Everett,  Massachusetts.  More  than  30  new  terminals  are  being  planned,  are  in  the 
licensing  process,  or  are  currently  under  construction  in  the  United  States. 

There  are  four  primary  obstacles  to  using  liquefied  natural  gas  (LNG)  to  address  energy 
needs  in  the  United  States:  1)  risk  of  fire  and  radiation  release  from  intentional  terrorist 
attacks  on  stationary  terminals  and  during  shipping,  2)  risk  of  fire  and  radiation  release 
from  unintentional  release  of  LNG,  3)  resistance  to  the  use  of  lowly  populated 
environmental  areas  for  industrial  purposes,  and  4)  reduced  property  values  to  local  areas. 

Terrorist  groups,  in  particular  The  Jihadist  Terrorist  network  of  al  Qaeda  and 
similar  groups,  have  the  intent  and  capability  to  attack  urban  LNG  facilities  and  tankers 
transporting  LNG.  The  goals  publicly  articulated  by  these  groups  include  1)  killing  large 
numbers  of  Americans,  2)  conducting  attacks  in  the  United  States,  and  3)  damaging  the 
US  economy,  and  4)  damaging  oil  and  gas  infrastructure.  The  capabilities  of  these 
groups  are  extensive,  as  illustrated  by  the  September  1 1th  attacks.  Those  attacks  required 
years  for  planning  and  reconnaissance,  as  well  as  specialized  knowledge  including  how 
to  fly  and  engage  in  close  quarter  combat.  They  demonstrated  the  ability  to  operate 
undetected  in  the  U.S.,  and  weapons  and  other  capabilities  to  attack  a  carrier  or  terminal 
are  readily  obtainable  in  the  US,  including  boats,  scuba  gear,  general  aviation  aircraft, 
fertilizer  based  and  commercial  explosives,  and  large  caliber  rockets  on  the  international 
gray  arms  market. 
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************  UNCLASSIFIED  ************** 


It  is  unlikely  that  deterrence  or  prevention  measures  would  be  adequate  to  defend  against 
attacks  from  skilled  terrorist  groups,  and  they  would  significantly  increase  costs. 
Nevertheless,  measures  to  reduce  the  risk  or  attacks  on  LNG  facilities  include  1)  locating 
LNG  terminals  more  than  three  miles  away  from  highly  populated  areas,  and  2) 
constructing  structures  around  LNG  facilities  similar  to  those  buildings  required  by  the 
Nuclear  Regulatory  Commission  around  commercial  nuclear  reactors.  To  reduce  the  risk 
of  attacks  during  shipping  transport,  measures  include  1)  locating  the  facility  away  from 
inland  waterway  transits,  2)  armor  plating  the  gas  storage  containers  aboard  the  LNG 
tankers,  and  3)  transporting  gas  in  tankers  that  do  not  freeze  and  condense  the  gas, 
thereby  reducing  the  force  and  radius  of  an  explosion. 

The  risk  of  an  unintentional  release  of  LNG  is  less  than  from  a  terrorist  attack  because  the 
LNG  release,  and  therefore  the  flammable  vapor  and  thermal  radiation  hazards,  would 
likely  be  far  less.  Nevertheless,  severe  negative  impacts  to  life  and  property  within  a 
couple  miles  could  result,  including  tens  of  thousands  of  deaths  and  injuries  due  to  severe 
burns  within  the  first  few  minutes  of  a  spill  of  LNG,  which  forms  a  floating  “pool  fire” 
when  ignited.  Comprehensive  evacuation  plans  and  education  of  surrounding 
populations  could  reduce  risks  by  delaying  going  outside  before  the  pool  fire  evaporates, 
thereby  reducing  unnecessary  exposure  to  high  levels  of  thermal  radiation. 

In  order  to  reduce  the  risks  of  intentional  or  unintentional  releases  of  LNG,  terminals  are 
often  recommended  to  be  located  several  miles  from  highly  populated  areas.  Particularly 
when  national,  state,  or  local  parks  are  recommended  locations,  there  is  resistance  to 
industrial  development.  The  ability  to  address  this  issue  depends  upon  the  availability 
and  accessibility  of  previously  developed,  lowly  populated  spaces  within  the  vicinity  of 
natural  gas  users. 

The  final  barrier  is  concern  about  reduction  of  property  values  due  to  proximity  to  an 
LNG  terminal.  Industrial  development  in  general  is  associated  with  a  decrease  in 
property  values  of  properties  within  a  two-mile  radius  due  to  visual  effects,  noise,  light, 
traffic  congestion,  and  odors.  Decreases  due  to  the  perceived  threat  of  terrorist  strikes  or 
unintentional  fires  are  also  possible.  On  the  other  hand,  property  values  near  existing 
LNG  terminals  have  not  significantly  decreased.  In  any  case,  the  perception  of  decreased 
property  values  by  local  residents  has  been  a  significant  obstacle  in  several  failed 
attempts  to  install  LNG  tenninals.  Measures  such  as  by  having  companies  provide 
protected  shoreline  access  to  private  properties  or  highlighting  increased  tax  revenue 
from  a  new  terminal,  may  address  this  obstacle  on  a  case  by  case  basis. 

This  analysis  does  not  include  the  front  end  of  the  LNG  supply  chain  (i.e.,  the 
exploration,  production,  and  liquefaction  of  gas  from  distant  and  isolated  locations)  or  the 
features  and  permitting  of  small  LNG  facilities  for  vehicle  fueling  or  peak-shaving 
purposes.  It  also  does  not  discuss  barriers  due  to  local  (i.e.,  state  or  regional)  legislation 
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************  UNCLASSIFIED  ************** 


Question:  What  are  the  primary  obstacles  to  using  Liquefied  Natural  Gas  (LNG)  to 
address  energy  needs  in  the  US? 

Natural  gas  is  steadily  becoming  the  U.S.  energy  source  of  choice.  It  now  supplies  about 
24  percent  of  the  United  States’  total  primary  energy  mix.  Natural  gas  is  also  among  the 
most  versatile  of  feedstocks  and  is  integral  to  industries  that  produce  plastics,  fertilizers, 
antifreeze  and  fabrics.  But  not  all  is  well  in  the  world  of  natural  gas  in  the  United  States, 
as  prices  have  risen  steadily  over  the  past  five  years.  In  times  past,  U.S.  natural  gas 
demand  was  entirely  satisfied  by  domestic,  and  then  Canadian,  production.  That  time  has 
ended.  Currently,  the  Department  of  Energy  estimates  the  United  States  has  about  1 89 
trillion  cubic  feet  of  proven  natural  gas  reserves  —  only  enough  to  supply  the  country’s 
needs  for  eight  years  at  current  consumption  rates.  If  a  long-term  solution  is  not  adopted 
quickly,  wide  swathes  of  U.S.  industry  will  simply  cease  functioning. 

U.S.  Natural  Gas  Prices  (wellhead) 


Luckily  for  the  United  States  liquefied  natural  gas,  or  LNG,  provides  a  way  to  both  plug 
the  gap  and  bring  down  prices  from  their  recent  highs.  Conventionally  piped  gas  changes 
from  its  gaseous  form  into  liquid  when  it  is  supercooled  to  around  -260  degrees 
Fahrenheit.  This  LNG  can  be  loaded  onto  specially  designed  tankers  and  shipped  in  a 
manner  similar  to  any  other  liquid.  Once  the  tanker  arrives  at  its  destination,  a  specialized 
facility  offloads  the  LNG,  at  which  point  it  can  be  loaded  into  any  infrastructure  that 
normally  stores,  transports  or  uses  conventional  natural  gas.  These  LNG  receiving 
terminals  can  be  placed  near  any  major  consumption  regions,  making  them  short-term  — 
and  economical  —  solutions  to  long-term  supply  problems. 

LNG  also  originates  from  states  that  are  stable  politically  and  economically.  The  barriers 
to  involvement  are  steep,  technical  and  expensive,  so  states  with  stability  problems  or 
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************  UNCLASSIFIED  ************** 


questionable  legal  regimes  simply  do  not  attract  the  necessary  interest.  Consequently,  the 
world’s  leading  LNG  providers  — Algeria,  Australia,  Qatar,  Oman  and  Trinidad  and 
Tobago  —  are  countries  that  largely  buy  into  the  U.S.  way  of  doing  things.  The  two 
notable  exceptions  are  Indonesia  and  Nigeria,  where  political  unrest  has  yet  to  scare  away 
what  have  been  the  world’s  most  successful  LNG  ventures  ever.  Other  states  that  the 
United  States  "trusts”  —  most  notably,  Egypt  and  Norway  —  also  are  joining  the  ranks  of 
LNG  producers.  Even  Russia  and  Libya  are  getting  into  the  act  in  a  limited  way. 

LNG  is  more  than  merely  cost  competitive.  It  is  far  cheaper  than  American  (or  Canadian) 
piped  natural  gas.  And  the  “global”  price  of  LNG  is  heading  down,  not  up.  Since  2001 
some  20  new  LNG  export  projects  have  begun,  the  first  of  which  came  on  line  in  just  the 
past  year.  Proven  reliable  suppliers  are  all  expanding  their  operations,  and  other  states  the 
United  States  considers  political  allies  are  also  attempting  to  cash  in  by  joining  the 
suppliers’  ranks.  All  told,  export  facilities  currently  under  construction  would  add  nearly 
3,000  billion  cubic  feet  of  supply  per  year,  but  import  facilities  currently  under 
construction  would  take  in  only  half  of  that.  The  result  will  be  a  glut  in  supply  that  will 
drive  domestic  prices  down  for  those  states  able  to  use  LNG. 

So  the  politics,  security  concerns  and  economics  of  LNG  and  the  United  States’  energy 
needs  match.  Unfortunately  for  the  United  States,  although  its  market-based  energy 
system  allows  for  efficient  supply  and  transport  of  energy,  the  country  lacks  a  unified 
energy  policy  capable  of  addressing  long-term  issues.  That  has  allowed  local  —  as 
opposed  to  national  —  environmental  groups  effectively  to  stall  the  development  of  LNG 
import  facilities. 

Though  national  groups  tacitly  approve  of  natural  gas,  and  thus  LNG,  local  and 
grassroots  groups  are  another  matter  entirely.  Most  local  groups  simply  do  not  care  about 
the  global  environmental  imperatives  dominating  the  national  groups’  agendas.  They 
instead  see  LNG  facilities  as  bombs  waiting  to  go  off.  National  groups  to  date  have  not 
taken  steps  to  rein  in  their  local  counterparts;  their  credibility  is  on  the  line.  The  net  effect 
is  that  they  are  paralyzed  and  cannot  say  what  they  know  to  be  true:  For  the  sake  of  the 
environment,  the  country  needs  more  LNG  importation  facilities. 

To  overcome  these  hurdles,  U.S.  President  George  W.  Bush  announced  he  would  seek  to 
extend  the  powers  of  the  Federal  Energy  Regulatory  Commission  (FERC)  so  that  it,  and 
not  the  various  states,  would  wield  final  authority  over  concerns  related  to  LNG  import 
developments.  If  that  power  is  granted,  the  FERC  would  be  able  to  override  local 
decision-making,  in  favor  of  constructing  LNG  import  terminals.  Expanding  FERC 
powers  will  not  defeat  local  environmental  groups  in  one  fell  swoop,  of  course.  But  the 
Bush  administration’s  new  policy  represents  the  inflection  point  in  the  LNG  debate.  It  is 
likely  only  a  matter  of  time  before  it  becomes  law  —  as  part  of  the  energy  bill  or 
independent  of  it  —  and  LNG  begins  streaming  to  the  United  States  in  massive  amounts. 
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************  UNCLASSIFIED  ************** 


PROCESS  DOCUMENTS  AVAILABLE 

The  following  documents  are  available,  each  of  which  details  a  part  of  the 
processes  that  were  perfonned  in  developing  the  two  written  analysis  reports: 

*  QUERY  SUMMARY 

*  DOCUMENTS  READ  (IN  ORDER  OF  READING) 

*  WHERE  DOCUMENTS  STORED  IN  FOLDERS 

*  KEY  DOCUMENTS  (RELIED  ON  HEAVILY) 

*  HYPOTHESES  —  CONSIDERED  OBSTACLES  TO  LNG 

*  COLLABORATIONS  WITH  OTHERS  DURING  PROCESS 

*  RESEARCH  NOTE-SHEET 


DOCUMENTS  AVAILABLE 


Figure  A.6:  Process  Documents  Available,  Page  1  of  1 
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************  UNCLASSIFIED  ************** 


QUERY  SUMMARY 

Queries  to  get  acquainted  with  area 

*  LNG  energy 

*  LNG  terrorism 

Targeted  search  for  documents: 

*  Sandia  report  LNG 

*  Clarke  LNG 

Targeted  search  to  investigate  claim  that  terrorism  concerns  are  a  shield  for 
concern  of  reduced  property  values  hy  local  population: 

*  Property  value  LNG 

Targeted  search  to  get  updated  numbers  on  LNG  terminals 

*  LNG  terminals  tankers 
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Figure  A.  7:  Analysis  Process  1  [A(1)]  Query  Summary,  Page  1  of  1 
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************  unclassified  ************** 


DOCUMENTS  READ  (IN  ORDER  OF  READING) 

1 .  Advertisement;  LNG  -  Evolution  &  Development  Wallchart  details  the  history 
of  the  LNG  industry  from  1914-2003;  www.biz-lib.com/ZPELE.html 

2.  California  energy  commission  website  page  on  Liquefied  Natural  Gas  (LNG); 
http://www.energy.ca.gov/lng/index.html 

3.  LNG:  A  Prized  Energy  Source,  or  Potent  Terror  Target?  -  Christian  Science 
Monitor  6apr04;  http://www.csmonitor.com/2004/0406/p01s01-uspo.html 

4.  Havens,  Jerry.  Ready  to  blow?  (Terrorism).  Bulletin  of  the  Atomic  Scientists, 
July,  2003;  http://www.thebulletin.org/article.php?art_ofn=ja03havens 

5.  Study:  LNG  tankers  make  spectacular  targets  for  terrorists.  Institute  for  the 
Analysis  of  Global  Security.  January  21,  2004.  http://www.iags.org/ 
n0121041.htm 

6.  Energy  security  and  liquefied  natural  gas.  Institute  for  the  Analysis  of  Global 
Security.  September  29,  2003.  http://www.iags.org/n0929034.htm 

7.  Consumer  Protection  Attorney  Tim  Riley  Homepage.  LNG  Terrorism  Danger 
To  Our  Communities.  http://timrileylaw.com/LNG_TERRORISM.htm 

8.  Scott  S.  Greenberger  and  Rick  Klein.  Officials  stop  gas  tanker  as  Boston 
reacts  to  alert.  Boston  Globe.  January  23,  2003.  http://www.boston.com/ 
news/local/articles/2003/ 12/23/ 
officials_stop_gas_tanker_as_boston_reacts_to_alert/ 

9.  LNG  Facilities  in  Urban  Areas:  A  Security  Risk  Management  Analysis  for 
Rhode  Island.  Richard  A.  Clarke,  May  2005;  http://www.riag.state.ri.us/ 

LN  G_Good%20Harbor2  .pdf 

10.  University  of  Houston  Law  Center,  Institute  for  Energy,  Law,  and  Enterprise, 
LNG  Safety  and  Security,  October  2003,  p.  77-79,  available  at  http:// 
www.energy.uh.edu/LNG/documents/IELE_LNG_Safety_and_Security.pdf 

1 1 .  Guidance  on  Risk  Analysis  and  Safety  Implications  of  a  Large  Liquefied 
Natural  Gas  (LNG)  Spill  Over  Water.  Sandia  National  Laboratories, 
December  2004.  http://www.pstrust.org/library/pdf/sandia_lng_1204.pdf 

12.  Statement  of  FERC  Chairman  Pat  Wood,  III,  on  2004  Sandia  report,  http:// 
www.ferc.gov/whats-new/headlines/2004/2004-4/12-22-04.pdf 
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************  UNCLASSIFIED  ************** 


13.  Battery  Rock  LNG:  A  Risk  Assessment.  GHC-BR-0306A.  Richard  A. 

Clarke,  March  2006. 

14.  Report  on  Potential  Economic  and  Fiscal  Impacts  on  the  Town  of  Harpswell, 
Maine  of  the  LNG  Terminal  Proposed  by  TransCanada  Pipelines  and 
ConocoPhillips.  Prepared  for  Fairplay  for  Harpswell  by  Yellow  Wood 
Associates,  Inc.,  228  North  Main  Street  St.,  Albans,  Vermont  05478;  http:// 
www.pstmst.org/lng/library/index.htm 

15.  U.S.  LNG  Markets  and  Uses.  Energy  Information  Administration  U.S. 
Department  of  Energy,  June  2004  Update;  http://www.pstmst.org/library/pdf/ 
inel_lngriskassess.pdf 

16.  The  Global  Liquefied  Natural  Gas  Market:  Status  &  Outlook 
December  2003  Energy  Infonnation  Administration  U.S.  Department  of 
Energy; 

http://www.pstmst.org/library/pdf/global_lng_market_2003.pdf 

17.  Liquefied  Natural  Gas  (LNG)  Import  Terminals:  Siting,  Safety  and  Regulation 
Prepared  by  Congressional  Research  Services  for  U.S.  Congress:  May  2004; 
http://www.pstmst.org/library/pdf/crs_report_may_2004.pdf 

18.  Liquefied  Natural  Gas:  History,  Risks  and  Siting:  The  California  Energy 
Commission's  white  paper.  July  2003;  http://www.pstmst.org/library/pdf/ 
cec_lng_white_paper.pdf 

19.  Spills  and  Fires  from  LNG  and  Oil  Tankers  in  Boston  Harbor 

by  MIT  Professor  James  A.  Fay:  77  Massachusetts  Avenue,  Rm.  3-258 
Cambridge,  MA02139  March  26,  2003;  http://www.pstmst.org/library/pdf/ 
fay_lng_fire_impact.pdf 

20.  Liquefied  Natural  Gas  (LNG)  Infrastructure  Security:  Background  and  Issues 
Prepared  by  Congressional  Research  Services  for  U.S.  Congress:  September 
2003.  http://www.pstmst.org/library/pdf/crs_rpt_lng_infra_security.pdf 

2 1 .  Qualitative  Risk  Assessment  of  LN G  Refueling  Station  and  Relevant  Safety 
Issues  Idaho  National  Engineering  Laboratory  report  1998.;  http:// 
www.pstmst.org/library/pdf/inel_lngriskassess.pdf 

22.  Girdis,  Dean.  Downeast  LNG  newsletter.  Volume  1,  Number  3.  November 
2005.  http://www.downeastlng.com/docs/November_Newsletter_No_3.pdf 
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************  UNCLASSIFIED  ************** 


23.  Finlaw,  James.  City  decries  LNG  plan.  Herald  News.  January  27,  2004. 
http://www.heraldnews.com/site/news.cfm? 

newsid=  10871 297&BRD=  1 7 1 0&PAG=46 1  &dept_id=99784&rfi=6 

24.  Save  Passamaquoddy  Bay:  A  3-Nation*  Alliance  to  Protect  the  Quoddy 
Region,  http://www.savepassamaquoddybay.org/ 

25.  California  energy  commission  website  page  on  Liquefied  Natural  Gas  (LNG); 
http://www.energy.ca.gov/lng/faq.html  (actually  the  second  visit  to  this  URL, 
but  doesn’t  realize  it) 

26.  West  Coast  LNG  Projects  and  Proposals.  Downloaded  from  California  energy 
commission  website.  Excel  spreadsheet  with  proposed  West  Coast  terminals. 
Status  update  as  of  March  13,  2006. 
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************  UNCLASSIFIED  ************** 

WHERE  DOCUMENTS  STORED  IN  FOLDERS 

LNG  obstacles  (no  subfolder) 

Advertisement;  LNG  -  Evolution  &  Development  Wallchart  details  the  history  of 
the  LNG  industry  from  1914-2003;  www.biz-lib.com/ZPELE.html 

California  energy  commission  website  page  on  Liquefied  Natural  Gas  (LNG); 
http  ://www.  energy,  ca.  go  v/lng/index.  html 

LNG:  A  Prized  Energy  Source,  or  Potent  Terror  Target?  -  Christian  Science 
Monitor  6apr04;  http://www.csmonitor.com/2004/0406/p0 1  sO  1  -uspo.html 

LNG  obstacles/Fire  Safety 

Guidance  on  Risk  Analysis  and  Safety  Implications  of  a  Large  Liquefied  Natural 
Gas  (LNG)  Spill  Over  Water.  Sandia  National  Laboratories,  December  2004. 
http://www.pstrust.org/library/pdf/sandia_lng_1204.pdf 

Statement  of  FERC  Chairman  Pat  Wood,  III,  on  2004  Sandia  report,  http:// 
www.ferc.gov/whats-new/headlines/2004/2004-4/12-22-04.pdf 

Liquefied  Natural  Gas  (LNG)  Infrastructure  Security:  Background  and  Issues 
Prepared  by  Congressional  Research  Services  for  U.S.  Congress:  September 
2003;  http://www.pstrust.org/library/pdf/inel_lngriskassess.pdf 

Qualitative  Risk  Assessment  of  LNG  Refueling  Station  and  Relevant  Safety 
Issues  Idaho  National  Engineering  Laboratory  report  1998.;  http:// 
www.pstrust.org/library/pdf/inel_lngriskassess.pdf 

Spills  and  Fires  from  LNG  and  Oil  Tankers  in  Boston  Harbor 
by  MIT  Professor  James  A.  Fay:  77  Massachusetts  Avenue,  Rm.  3-258 
Cambridge,  MA  02139  March  26,  2003;  http://www.pstrust.org/library/pdf/ 
fay_lng_fire_impact.pdf 


LNG  obstacles/Imports 

The  Global  Liquefied  Natural  Gas  Market:  Status  &  Outlook 

December  2003  Energy  Information  Administration  U.S.  Department  of  Energy; 

http://www.pstrust.org/library/pdf/global_lng_market_2003.pdf 
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************  unclassified  ************** 


LNG  obstacles/LNG  Background 

U.S.  LNG  Markets  and  Uses.  Energy  Information  Administration  U.S. 
Department  of  Energy,  June  2004  Update;  http://www.pstrust.org/library/pdf/ 
inel_lngriskassess.pdf 

Liquefied  Natural  Gas:  History,  Risks  and  Siting:  The  California  Energy 
Commission's  white  paper.  July  2003;  http://www.pstmst.org/library/pdf/ 
cec_lng_white_paper.pdf 

California  energy  commission  website  page  on  Liquefied  Natural  Gas  (LNG); 
http  ://www.  energy,  ca.  go  v/lng/  faq.html 

West  Coast  LNG  Projects  and  Proposals.  Downloaded  from  California  energy 
commission  website.  Excel  spreadsheet  with  proposed  West  Coast  terminals. 
Status  update  as  of  March  13,  2006. 

LNG  obstacles/Propertv  Values 

Save  Passamaquoddy  Bay:  A  3-Nation*  Alliance  to  Protect  the  Quoddy  Region. 
http://www.savepassamaquoddybay.org/ 

Girdis,  Dean.  Downeast  LNG  newsletter.  Volume  1,  Number  3.  November  2005. 
http://www.downeastlng.com/docs/November_Newsletter_No_3.pdf 

Report  on  Potential  Economic  and  Fiscal  Impacts  on  the  Town  of  Harpswell, 
Maine  of  the  LNG  Terminal  Proposed  by  TransCanada  Pipelines  and 
ConocoPhillips.  Prepared  for  Fairplay  for  Harpswell  by  Yellow  Wood  Associates, 
Inc.,  228  North  Main  Street  St.,  Albans,  Vermont  05478;  http://www.pstmst.org/ 
lng/library/index.htm 

Finlaw,  James.  City  decries  LNG  plan.  Herald  News.  January  27,  2004.  http:// 
www.heraldnews.com/site/news.cfm? 

newsid=  10871 297&BRD=  1 7 1 0&PAG=46 1  &dept_id=99784&rfi=6 

LNG  Obstacles/Regulations 

Liquefied  Natural  Gas  (LNG)  Import  Tenninals:  Siting,  Safety  and  Regulation 
Prepared  by  Congressional  Research  Services  for  U.S.  Congress:  May  2004; 
http://www.pstmst.org/library/pdf/crs_report_may_2004.pdf 


PROCESS  #1 


Figure  A.  12:  Analysis  Process  1  [A(1)]  Where  Documents  Stored  in  Folders,  Page  2  of  3 


(Appendix  A) 


73 


************  UNCLASSIFIED  ************** 


LNG  Obstacles/Terrorism 

Havens,  Jerry.  Ready  to  blow?  (Terrorism).  Bulletin  of  the  Atomic  Scientists, 
July,  2003 ;  http://www.thebulletin.org/article.php?art_ofn=ja03havens 

Study:  LNG  tankers  make  spectacular  targets  for  terrorists.  Institute  for  the 
Analysis  of  Global  Security.  January  21,  2004.  http://www.iags.org/ 
n0121041.htm 

Energy  security  and  liquefied  natural  gas.  Institute  for  the  Analysis  of  Global 
Security.  September  29,  2003.  http://www.iags.org/n0929034.htm 

Consumer  Protection  Attorney  Tim  Riley  Homepage.  LNG  Terrorism  Danger  To 
Our  Communities.  http://timrileylaw.com/LNG_TERRORISM.htm 

Scott  S.  Greenberger  and  Rick  Klein.  Officials  stop  gas  tanker  as  Boston  reacts  to 
alert.  Boston  Globe.  January  23,  2003.  http://www.boston.com/news/local/ 
articles/2003/12/23/officials_stop_gas_tanker_as_boston_reacts_to_alert/ 

LNG  Facilities  in  Urban  Areas:  A  Security  Risk  Management  Analysis  for  Rhode 
Island.  Richard  A.  Clarke,  May  2005;  http://www.riag.state.ri.us/LNG_Good% 
20Harbor2.pdf 

Battery  Rock  LNG:  A  Risk  Assessment.  GHC-BR-0306A.  Richard  A.  Clarke, 
March  2006. 
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************  UNCLASSIFIED  ************** 


KEY  DOCUMENTS  (RELIED  ON  HEAVILY) 

Terrorism  with  LNG  terminals  and  tankers: 

LNG  Facilities  in  Urban  Areas:  A  Security  Risk  Management  Analysis  for  Rhode 
Island.  Richard  A.  Clarke,  May  2005;  http://www.riag.state.ri.us/LNG_Good% 
20Harbor2.pdf 

Safety  (unintentional  release  of  l.NG): 

Guidance  on  Risk  Analysis  and  Safety  Implications  of  a  Large  Liquefied  Natural 
Gas  (LNG)  Spill  Over  Water.  Sandia  National  Laboratories,  December  2004. 
http://www.pstrust.org/library/pdf/sandia_lng_1204.pdf 

Environment  (impact  of  facility  on  local  population!: 

Liquefied  Natural  Gas  (LNG)  Import  Terminals:  Siting,  Safety  and  Regulation 
Prepared  by  Congressional  Research  Services  for  U.S.  Congress:  May  2004; 
http://www.pstrust.org/library/pdf/crs_report_may_2004.pdf 

Girdis,  Dean.  Downeast  LNG  newsletter.  Volume  1,  Number  3.  November  2005 
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************  unclassified  ************** 


HYPOTHESES  -  CONSIDERED  OBSTACLES  TO  LNG 

HI:  Terrorism 

H1A:  Terminals 
H1B:  Tankers 
H 1 C :  Trucks  ( dropped) 

HID:  Pipelines  (never  seriously  pursued) 

H2:  Safety  (unintentional  LNG  release) 

H2A:  Terminals 
H2B:  Tankers 

H3:  Environment 

H3A:  Property  values 

H3B:  Development  of  lowly  populated  areas  such  as  national  parks 
H3C:  Pollution  from  facility  (dropped) 

H3D:  Economic  impacts  (dropped) 

H4:  Importing  from  foreign  countries  (dropped) 

H5:  Regulations  (judged  to  be  out  of  scope) 
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Figure  A.  15:  Analysis  Process  1  [A(l)]  Hypotheses,  Page  1  of  1 
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************  UNCLASSIFIED  ************** 


COLLABORATIONS  WITH  OTHERS  DURING  PROCESS 

1 .  Customer  who  Asked  Question:  Clarifies  question  via  telephone  call 

2.  Analyst  Colleague:  Frames  LNG  issues  through  e-mail  exchange 

3.  Specialist  Analyst:  Consults  with  with  in-house  counter-terrorism  experts  on 
limited  topics  through  e-mail  and  in-person  interactions 

4.  Industry  Consultant:  Requests  more  information  and  context  behind  policy¬ 
making  suggestions  noted  in  reports 

5.  LNG  Researcher:  Explores  divergent  views  regarding  potential  impacts  of 
LNG  release  via  telephone  conversation 

6.  “Biased  Study”  Representative:  Confirms  that  study  results  were  biased  due  to 
desire  to  influence  a  policy  decision 

7.  Senior  Analyst:  Review  of  report  results  in  the  addition  of  a  fourth  obstacle 
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************  UNCLASSIFIED  ************** 


RESEARCH  NOTE-SHEET 

Tankers 

http://www.csmonitor.com/2004/0406/p01s01-uspo.html 
Energy  analysts  call  LNG  the  "new  prize"  on  the  global  energy  scene.  Japan  and 
other  energy-poor  nations  have  long  imported  large  amounts  of  LNG.  The  US 
expansion  is  part  of  a  global  boom,  with  at  least  55  new  LNG  tankers  under 
construction  -  a  one-third  increase  in  the  world  fleet  to  more  than  200  vessels. 

Past  LNG  explosions  (not  NG  explosions) 

*  Algeria  January  2004:  22  killed,  74  injured. 

Energy  companies 

*  Leading  the  charge  are  big  energy  companies  such  as  BP  and  ExxonMobil  - 
along  with  smaller  concerns  like  Weaver's  Cove  Energy,  which  wants  to  build 
a  terminal  in  Fall  River.  They  argue  that  fears  of  terrorism  and  even  human 
error  in  handling  the  fuel  are  vastly  overblown,  and  point  to  strong  safety 
records,  tighter  post-9/1 1  security,  and  robust  LNG  tankers.  Two  controversial 
studies  since  9/11,  one  for  the  US  Department  of  Energy  (DOE)  and  another 
for  the  industry,  suggest  LNG  tanker  fires  would  be  a  smaller  threat  than  many 
fear. 

*  "Look  at  the  existing  safety  record  [of  LNG  tankers]  -  it's  sterling,  extremely 
clean,"  says  Martin  Edwards  of  the  Interstate  Natural  Gas  Association  of 
America,  a  trade  group  representing  companies  with  current  and  proposed 
LNG  terminals.  "What  we  often  find  is  that  the  safety,  and  lately  the  security, 
issues  are  a  kind  of  shield"  that  masks  opponents'  deeper  concerns,  such  as 
threats  to  property  values. 

*  A  plan  to  build  a  terminal  at  the  Port  of  Long  Beach  is  among  the  furthest 
along.  "This  is  going  to  be  the  safest  LNG  receiving  terminal  in  the  world," 
Tom  Giles,  chief  of  Sound  Energy  Solutions,  a  Mitsubishi  subsidiary,  told  the 
Los  Angeles  Times  in  January  when  describing  the  concrete  and  steel  tanks  it 
planned  to  build. 

Number  of  terminals  and  tankers 

*  From  2004  report  -  need  to  update  figures:  Today  there  are  four  LNG 
tenninals  in  the  US:  Everett,  Mass.,  near  Boston;  Cove  Point,  Md.;  Elba 
Island,  Ga.;  and  Lake  Charles,  La.  Because  US  supplies  of  natural  gas  are 
in  short  supply,  more  than  30  LNG  terminals  are  under  consideration, 
including  some  near  densely  populated  areas  like  Fall  River,  Mass.,  Long 
Beach,  Calif.,  Logan  Township,  N.J.,  and  Providence,  R.I. 

*  In  addition  to  the  Everett  facility  there  are  operational  plants  at  Cove  Point 
in  the  Chesapeake  Bay,  Maryland,  in  Savannah,  Georgia,  and  in  Lake 
Charles,  Louisiana. 

PROCESS  #1 


Figure  A.  17:  Analysis  Process  1  [A(l)]  Research  Note-Sheet,  Page  1  of  17 


(Appendix  A) 


78 


************  UNCLASSIFIED  ************** 


Trusted  to  be  uptodate  on  March  14,  2006  from  http://www.energy.ca.gov/lng/ 
intemational.html: 

Natural  gas,  one  of  the  largest  sources  of  energy,  can  be  found  all  over  the  world. 
LNG  comes  from  countries  with  large  natural  gas  reservoirs  including  Algeria, 
Australia,  Brunei,  Indonesia,  Libya,  Malaysia,  Nigeria,  Oman,  Qatar,  and  Trinidad 
and  Tobago.  The  largest  gas  reserves  can  be  found  in  the  Middle  East.  Much  of 
the  natural  gas  reserves  found  around  the  world  are  separate  from  oil  and  as  new 
reserves  are  discovered  and  processed,  growth  in  the  LNG  industry  will  continue. 
Worldwide  there  are  currently  12  countries  that  export  LNG.  There  are 
approximately  40  LNG  import  terminals  with  many  more  planned.  LNG  import 
terminals  exist  in  Japan,  South  Korea  and  Europe,  as  well  as  in  the  United  States, 
which  currently  has  five  import  terminals  (including  Puerto  Rico). 
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History  of  LNG 

•  Evolution  &  Development  Wallchart  details  the  history  of  the  LNG  industry 
from  1914-2003,  providing  more  extensive  detail  on  recent  developments 

•  Tables  list  existing  LNG  export  plants  (country,  project,  start-up  year,  share 
holders,  operator,  capacity  (m  t/y),  no.  of  trains  )  and  import  terminals 
(country,  project,  start-up  year,  promoter,  operator,  source,  capacity  (m  t/y) 

•  Chart  showing  LNG  exports  and  imports  by  country  (Bn  cm)  from  1964-2002 

The  last  time  there  was  so  much  interest  in  LNG  was  in  the  1970s,  when  the  four 
current  continental  U.S.  LNG  import  terminals  were  built.  At  that  time  three 
terminals  were  also  proposed  for  southern  California  alone— at  Los  Angeles,  Point 
Conception,  and  Oxnard.  From  environmental  and  health  impacts  to  the 
consequences  of  a  catastrophic  event,  each  site  was  evaluated  on  the  basis  of 
public  safety.  The  studies  identified  four  areas  of  concern  for  catastrophic 
accident  events: 

•  vapor  cloud  explosions, 

•  pool  fires  on  the  surface  of  the  water  or  on  land, 

•  flammable  vapor  clouds  that  can  form  if  a  spill  does  not  ignite  immediately,  and 

•  rapid  phase  transition  (RPT)  accidents  from  rapid  mixing  of  LNG  with  water. 

The  three  California  projects  were  shelved  for  economic  reasons  along  with  their 
safety  studies,  but  not  before  raising  public  concern.  As  a  result.  Congress 
authorized  approximately  $40  million  to  study  the  four  catastrophic  scenarios. 

Rapid  phase  transitions.  What  can  happen  when  LNG  mixes  with  water  is  the 
least  studied  of  the  four  hazards,  especially  at  large  scale.  The  hazard  zones 
extending  from  an  RPT  would  not  be  as  large  as  either  vapor  cloud  or  pool  fire 
hazard  zones.  RPT  accidents  are  probably  the  lowest  concern  of  the  four 
mentioned  here,  but  the  potential  for  secondary  damage  resulting  from  an  RPT 
accident  should  also  be  considered. 

Flammable  vapor  clouds.  Most  experts  believe  that  an  accidental  release  of  an 
entire  LNG  ship  storage  tank,  such  as  might  occur  in  a  collision,  would  result  in 
the  immediate  ignition  of  the  released  gas.  This  would  prevent  the  development  of 
extremely  large,  and  potentially  more  catastrophic,  vapor  cloud  fires.  Nonetheless, 
current  regulations  require  that  this  issue  be  addressed. 

Vapor  cloud  explosions.  LNG  is  usually  at  least  90  percent  methane.  Experiments 
have  shown  that  if  ignited,  uncontained  methane-air  clouds  could  cause 
catastrophic  fire  damage,  but  they  have  low  explosion  potential.  Research 
continues  on  the  effects  of  partial  confinement  and  turbulence  in  such  clouds,  but 
the  results  so  far  suggest  that  the  vapor  cloud  explosion  scenario  is  of 
considerably  less  concern  than  the  fire  scenario. 

Pool  fires.  In  my  judgment,  a  large  LNG  pool  fire— on  water,  and  therefore 
uncontained— is  of  the  highest  concern. 
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What  do  we  know  about  the  hazards  of  such  pool  fires?  There  have  been  no 
experiments  with  fires  of  such  magnitude.  By  marine  cargo  standards,  LNG 
marine  carriage  is  relatively  new,  and  there  is  no  evidence  of  any  pool  fires  on 
water  larger  than  experimental  spills  of  about  10,000  gallons  performed  during 
the  1970s  and  1980s.  These  experiments  resulted  in  roughly  cylindrical  fires 
approximately  50  feet  wide  and  250  feet  high. 

Although  large-model  extrapolations  (from  10,000  gallons  to  6.5  million  gallons) 
can  raise  significant  questions,  it  is  reasonable  to  be  concerned  about  the  damage 
potential  of  such  fires.  Most  predictions  suggest  that  even  the  largest  LNG  tankers 
(typically  more  than  900  feet  in  length)  might  be  completely  enveloped  in  a  pool 
fire  following  a  complete  spill  of  a  single  6.5  million  gallon  tank.  This  raises 
questions  about  the  vulnerability  of  the  ship  and  the  potential  for  additional 
releases.  A  typical  LNG  tanker  contains  as  many  as  five  tanks  with  a  combined 
capacity  of  33  million  gallons. 

We  do  know  some  things  about  such  fires.  They  could  not  be  extinguished  and 
would  have  to  burn  themselves  out.  Unlike  some  other  flammable  liquids  such  as 
crude  oil,  the  gas  would  burn  itself  out  only  when  it  was  totally  consumed.  And 
such  fires  would  be  expected  to  bum  more  rapidly  and  with  greater  intensity  than 
cmde  oil  or  even  gasoline  fires. 
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Obstacles  to  LNG: 

o  Terrorism 

■  Tankers 

2003  article:  There  are  only  four  LNG  import  tenninals  in  the  United  States: 
Chesapeake  Bay,  Maryland;  Lake  Charles,  Louisiana;  Elba  Island,  Georgia;  and 
Everett,  Massachusetts.  Tankers  traveling  to  Everett,  in  Boston  Harbor,  pass  near 
heavily  populated  areas.  Along  both  inbound  and  outbound  routes,  LNG  ships 
travel  within  several  hundred  yards  of  the  Boston  waterfront,  past  the  end  of  the 
Logan  International  Airport  runway  from  which  two  planes  left  for  the  World 
Trade  Towers,  and  under  a  busy  bridge.  Even  now,  ships  coming  into  Everett  are 
subjected  to  greater  scrutiny  than  before  September  1 1 . 

In  an  interview  with  Energy  Security  Fay  said  a  terrorist  attack  by  a  boat  bomb  - 
such  as  the  one  used  against  the  USS  Cole  in  2000  or  the  French  tanker  Limburg 
off  the  coast  of  Yemen  in  2002  -  could  cause  at  least  half  a  cargo  hold's  worth  of 
LNG  to  seep  out  of  the  ship  and  ignite.  "In  just  over  three  minutes,  the  fire  could 
spread  two-thirds  of  a  mile  from  the  ship,"  Fay  said.  "There  is  nothing  safety 
officials  can  do  in  such  a  case.  They  would  have  no  time  to  evacuate  people  or  to 
put  out  the  fire.”  Fay  also  predicts  damaging  thermal  radiation  within  a  mile 
radius  of  the  tanker  which  could  set  fire  to  thousands  of  homes  and  cause 
significant  losses  of  blood  and  treasure.  "Like  the  attack  on  the  World  Trade 
Center  in  New  York  City,  there  exists  no  relevant  industrial  experience  with  fires 
of  this  scale  from  which  to  project  measures  for  securing  public  safety,"  he  says. 
Fay  insists  the  methodology  of  his  modeling  is  sound. 

LNG  tankers  are  very  conspicuous.  Their  distinctive  storage  tanks  jut  like  humps 
on  the  decks;  their  identity  cannot  be  mistaken.  Terrorists  attempting  to  target 
such  a  ship  will  have  no  problem  identifying  it. 

Boston  Mayor  Thomas  Menino  recently  decided  to  rid  Boston  Harbor  of  its  long¬ 
standing  LNG  facility  over  safety  concerns.  "Everyone  should  be  concerned  about 
it  because  the  Coast  Guard,  Boston  fire  department  and  other  agencies  do  not 
have  the  equipment  if  something  did  happen  with  an  LNG  tanker.  Everyone  says 
there  is  no  problems,  but  what  happens  when  something  does  happen?"  Menino 
said  this  past  December  when  the  national  threat  level  was  elevated  to  orange. 
Menino  and  other  representatives  of  Boston-area  communities  had  mounted  an 
unsuccessful  lawsuit  to  halt  the  LNG  operations  after  Sept.  11,  2001.  Professor 
Fay  agrees.  "Federal  officials  are  at  a  state  of  denial  right  now.  They  ignore  the 
scenario  of  tanker  spill  as  a  problem  they  have  to  deal  with."  Menino  has  no 
jurisdiction  in  the  harbor  so  the  tankers  are  still  coming. 

The  Coast  Guard  however  is  not  ignoring  the  threat.  It  has  taken  some  precautions 
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to  minimize  the  risk  of  attack  against  LNG  tankers.  Fast  escort  boats  shepherd 
each  gas  tanker  as  it  travels  to  the  terminal.  A  security  zone  extending  500  yards 
on  each  side,  two  miles  ahead  and  a  mile  behind  the  tanker  is  imposed  and  other 
vessels  are  instructed  to  give  the  tanker  a  wide  berth  during  its  passage  and  12- 
hour  unloading  process.  Violators  face  arrest,  fines  of  up  to  $25,000  and  prison 
terms  of  up  to  10  years.  But  these  penalties  are  unlikely  to  deter  suicide  terrorists 
such  as  those  who  flew  planes  into  the  World  Trade  Center  and  the  Pentagon.  It  is 
not  clear  what  procedures  the  Coast  Guard  would  be  willing  to  use  once  a  terrorist 
boat  penetrates  into  the  security  zone.  Nor  it  is  clear  how  rapidly  security  officials 
could  respond  to  the  threat.  After  all,  well  armed  and  vigilant  military  targets  like 
the  USS  Cole  could  not  prevent  such  an  attack. 

Intent  of  terrorists  on  tankers 

As  to  intent  to  attack  shipping,  the  al  Qaeda  network  has 
used  explosive  laden  small  craft  to  attack  a  US  destroyer  in 
port  and  a  double  hulled  French  tanker  at  sea.  They  have 
planned  or  discussed  attacks  on  shipping  in  other  locations 
around  the  world.  The  FBI  has  warned  that  the  al  Qaeda 
network  is  interested  in  scuba  gear  for  underwater  attacks 
in  the  US. 

In  a  recently  released  document  known  simply  as  the  National 
Planning  Scenarios,  DHS  indicated  that  a  potential  terrorist 
attack  on  chemical  or  gas  tanker  is  the  number  six  ranked 
doomsday  scenario  for  the  United  States  government.  As  a 
result,  DHS  is  expected  to  spend  at  least  an  additional  one 
billion  dollars  to  secure  against  this  fonn  of  terrorist 
attack.  However  even  those  within  DHS  believe  that  the 
United  States  is  a  long  way  away  from  true  preparedness.33 

Currently,  over  80%  of  the  United  States  natural  gas  imports 
are  shipped  in  tankers  from  Trinidad  and  Tobago,  which  are 
attractive  targets  to  terrorist  organizations.34  As  natural 
gas  demands  increase  in  the  United  States,  natural  gas- 
producing  countries  will  increase  their  export. 

Interestingly,  the  same  countries  that  currently  provide 
much  of  the  US’s  current  oil  supply  will  mostly  likely  be 
the  same  increasing  their  production  of  LNG:  namely 
countries  in  West  Africa  and  the  Persian  Gulf  -  areas  where 
al  Qaeda  has  an  already  established  a  foothold. 
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Capability  of  terrorists  on  tankers  and  terminals 
A1  Qaeda  and  related  groups  have  demonstrated 
an  ability  to  operate  in  the  US.  Even  since  9-11,  terrorist 
groups  have  maintained  a  presence  in  the  US.  A  recent 
report  indicated  that  the  FBI  has  over  1000  Full  Field 
Investigations  underway  against  al  Qaeda  alone.  Illegal 
crossing  into  the  United  States  is  a  commonplace  activity. 

Weapons  and  other  capabilities  needed  to  conduct  an  attack 
on  an  urban  LNG  offloading  facility  or  an  LNG  tanker  can  be 
readily  obtained  in  the  US,  according  to  US  Government 
reports.  A  variety  of  boats  and  scuba  gear  can  be  easily 
procured.  General  Aviation  aircraft  can  easily  be  rented  or 
stolen  at  numerous  small  airports  throughout  the  US. 
Explosives  are  readily  available,  both  fertilizer  based 
weapons,  which  can  be  procured  without  a  license,  and 
commercial  explosives,  which  are  frequently  stolen  and  sold 
on  the  black  market.  Fifty  caliber  rifles  with  anti-annor 
shells  are  readily  available  in  the  US.  Rocket  propelled 
grenades  (RPGs),  light  anti-tank  weapons,  mortars,  and 
bazooka  styled  weapons  are  very  easily  and  cheaply 
obtainable  on  the  international  gray  arms  market.  Few 
containers  entering  the  United  States  are  inspected  by  US 
Customs. 


Vulnerabilities  to  tankers 

As  to  the  LNG  ship,  the  creation  of  restricted  waterways 
around  the  LNG  tanker  and  the  use  of  armed  Coast  Guard 
(USCG)  patrol  craft  provides  little  assurance  that  a 
determined  terrorist  group  would  be  stopped  before  attacking 
the  tanker  with  an  explosives  laden  vessel.  Narraganset  Bay 
is  home  to  thousands  of  small  craft.  The  USCG  and  other  law 
enforcement  agencies  would  be  reluctant  to  use  lethal  force 
against  an  apparently  misguided  pleasure  craft.  Moreover, 
the  escorting  patrol  boats  could  themselves  be  attacked  in  a 
multi-boat  terrorist  operation.  Counter-SCUBA  operations  in 
the  Bay  would  also  not  offer  high  assurance  of  success. 

Attacks  involving  stand  off  weapons  could  be  mounted  from 
boats  or  from  numerous  land  locations  along  the  route.  To 
prevent  the  entry  of  weapons  for  land  based,  stand-off 
attacks,  all  vehicles  entering  the  littoral  would  have  to  be 
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searched  not  just  during  the  tanker’s  transit,  but  at  all 
times. 

■  Terminals 


Intent  of  terrorists 

The  Jihadist  Terrorist  network  of  al  Qaeda  and 
similar  groups  have  articulated  goals  including  a)  killing 
large  numbers  of  Americans,  b)  conducting  attacks  in  the  US, 
c)  damaging  the  US  economy  and  infrastructure,  and  d) 
damaging  oil  and  gas  infrastructure. 

The  al  Qaeda  network  has  demonstrated  the  use  of  parts  of 
the  US  civilian  infrastructure  as  weapons  to  be  used  against 
US  facilities. 

Furthermore  LNG  installations  can  be  attacked  onshore  by  truck  bombs  with 
similarly  damaging  consequences. 

There  are  two  reasons  why  a  terrorist  would  be  interested  in 
attacking  an  LNG  tanker  or  facility,  both  of  which  fit  al 
Qaeda’s  MO:  the  potential  for  high  civilian  casualties  and 
the  potential  to  bring  substantial  damage  the  American 
economy.  Although  the  United  States  acquires  roughly  2%  of 
its  overall  gas  consumption  from  LNG  sources,  some  analysts 
predict  that  this  amount  is  likely  to  increase  at  a  rate  of 
2%  a  year.3 1  As  LNG  imports  become  a  more  important  sector  of 
our  economy,  terrorist  organizations  like  al  Qaeda  will 
become  more  interested  in  attacking  them.  In  addition,  LNG 
tankers,  which  often  travel  in  close  proximity  to 
metropolitan  seaports,  are  undoubtedly  attractive  high 
casualty  targets  for  al  Qaeda  planners. 

In  addition  to  its  interests  in  attacking  tankers,  al  Qaeda 
would  have  the  motivation  and  could  develop  the  capability 
to  attack  LNG  facilities.  MS-13,  a  Central  American 
criminal  organization  with  a  large  membership  in  East 
Boston,  is  feared  to  be  targeting  the  LNG  facility  and 
tankers  near  Boston.  MS- 13  has  a  strong  presence  in 
harborside  neighborhoods  of  East  Boston  alongside  which  LNG 
tankers  pass  on  their  way  to  the  unloading  facility  in 
Everett.  In  January,  these  members  were  subject  to  a 
Homeland  Security  and  Customs  Department  investigation  after 
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the  Justice  Department  announced  that  al  Qaeda  operatives 
might  be  trying  to  get  into  the  country  through  Mexico.  MS- 
1 3  is  believed  to  be  involved  in  smuggling  from  Mexico  to 
the  U.S.35  Though  al  Qaeda  has  yet  to  attempt  an  attack  on 
the  LNG  facility  in  Boston,  to  most  counterterrorism 
officials,  the  threat  is  clear  and  present. 

Vulnerabilities 

Both  the  proposed  urban  LNG  off  loading 
facility  and  the  proposed  LNG  tanker  transit  through  29 
miles  of  Rhode  Island  have  security  vulnerabilities  that  are 
unlikely  to  be  successfully  remediated. 

The  creation  of  permanent  or  temporary  restricted  flight 
areas  around  the  urban  LNG  facility  and  the  tanker  will  not 
prevent  hijacked  or  stolen  aircraft  (commercial  passenger, 
commercial  freight,  or  general  aviation)  from  successfully 
penetrating  the  restricted  airspace  and  crashing  into  the 
facility  and/or  ship.  No  air  defense  system  is  planned,  nor 
is  it  easy  to  imagine  a  system  which  would  authorize  the  use 
of  deadly  force  against  an  aircraft  that  might  appear  to 
have  unintentionally  strayed  into  the  restricted  air  space. 

*  Jan  21,  2004:  The  safety  concerns  surrounding  LNG  installations  pose 
difficulties  for  energy  companies  attempting  to  build  new  terminals.  No 
such  terminals  have  been  built  in  the  U.S.  for  two  decades,  but 
applications  to  construct  30  more  have  been  made  in  recent  years.  Only 
half  a  dozen  are  likely  to  materialize  in  the  next  decade.  ExxonMobil  has 
announced  plans  to  build  a  $600  million  plant  on  the  Texas  coast  and 
wants  to  build  three  more  in  other  states.  ChevronTexaco  announced  plans 
to  construct  an  off  the  coast  of  Baja  California,  Mexico  and  Royal  Dutch/ 
Shell  and  BP  are  among  other  companies  driving  to  build  new  terminals  in 
California,  Texas,  Alabama,  Florida,  Mexico,  Nova  Scotia  and  other 
locations.  In  most  of  these  places  opposition  by  local  communities  is 
mounting  and  it  is  not  yet  clear  which  consideration  will  prevail:  public 
safety  or  economic  need. 

■  Trucks 
o  Fire  Safety 

But  Sandia  says  the  consequences  of  an  intentional  breach, 
in  which  the  LNG  release  would  likely  be  far  greater,  could 

PROCESS  #1 


Figure  A. 25:  Analysis  Process  1  [A(1)]  Research  Note-Sheet,  Page  9  of  77 


(Appendix  A) 


86 


************  unclassified  ************** 


have  “severe  negative  impacts”  in  its  damage  to  bridges, 
industrial/commercial  centers,  LNG  terminals,  harbors,  and 
populated  areas.  In  its  DEIS,  FERC  tends  to  downplay  the 
terrorist  threat  to  the  LNG  industry  in  the  United  States. 

It  is  reluctant  to  acknowledge  the  potential  for  large-scale 
disaster  should  a  worst-case  scenario  LNG  release  result 
from  a  deliberate  attack  on  a  tanker  or  a  facility.  FERC 
concludes  its  analysis  of  the  terrorist  threat  by  shifting 
the  focus  of  the  discussion  away  from  LNG  to  other  potential 
terrorist  targets  in  the  U.S.  “At  the  national  level,"  the 
DEIS  says,  “potential  terrorist  targets  are  plentiful,  many 
having  national  significance,  while  others  with  a  large 
concentration  of  the  public  (major  sporting  events, 
skyscrapers,  etc,)  or  critical  infrastructure  facilities.” 

FERC  points  out  that  the  U.S.  currently  has  over  500 
chemical  facilities  operating  near  large  populations,  with 
over  1 00,000  shipments  of  hazardous  cargo  being  shipped 
through  U.S.  waterways  each  year.  FERC  says  that  "resources 
can  be  directed  to  mitigate  possible  attack  paths”  for 
potential  targets  where  the  threat  is  perceived  to  be  high, 
and  that  decision  makers  must  determine  “whether  the 
resources  required  to  manage  the  risks  are  justified  by  the 
benefits”  provided  by  the  potential  target  in  question. 

o  Imports 

o  Property  values 

Consequences  of  attack  (fire) 

There  is  a  spectrum  of  expert  opinion  on 
the  precise  extent  of  damage  that  would  result  from  various 
levels  of  attack  on  an  urban  LNG  facility  and  on  an  LNG 
tanker.  There  appears,  however,  to  be  a  high  risk  that 
catastrophic  damage  could  occur  if  a  large  breach  were  made 
in  the  urban  LNG  facility’s  tank,  if  three  of  five 
containers  aboard  the  LNG  tanker  were  breached,  or  if  an 
attack  occurred  involving  both  the  facility  and  the  tanker 
during  unloading. 

The  consequences  of  a  major  attack  could  include  fires  that 
would  damage  homes,  hospitals,  a  chemical  plant,  and  other 
infrastructure,  depending  upon  where  the  attack  occurred. 
Many  fires  could  exceed  the  2000  BTU  limit  for  the 
employment  of  fire  fighters,  necessitating  a  “let  it  burn” 
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approach  to  many  structures.  There  would  be  both  prompt  and 
delayed  fatalities. 

The  delayed  fatalities  and  the  wounded  could  place  a  burden 
on  the  Rhode  Island  and  South  Eastern  Massachusetts  trauma, 
bum,  and  overall  emergency  medical  response  capability  that 
the  system  would  be  unable  to  handle.  It  is  unclear  where 
the  funding  would  come  from  to  upgrade  the  region’s 
consequence  management  capabilities  to  be  able  to  deal  with 
a  possible  catastrophic  attack  on  the  urban  LNG  facility 
and/or  tanker.  Governments  could,  however,  place  that 
burden  on  the  facility  owners  and  operators,  similar  to  the 
Nuclear  Regulatory  Commission’s  approach  to  commercial 
nuclear  reactors. 

Financial  cost 

The  financial  cost  of  compensating  victims 
and  rebuilding  damaged  or  destroyed  facilities  following  a 
catastrophic  attack  on  the  urban  LNG  facility  and/or  LNG 
tanker  would  likely  exceed  any  insurance  carried  by  the 
owners  and  operators  of  the  LNG  facility  and  tanker. 

In  the  absence  of  adequate  insurance  to 
pay  victims  and  rebuild  damaged  or  destroyed  facilities,  the 
LNG  operators  would  be  transferring  the  financial  cost  of 
the  risk  they  would  be  creating  either  to  the  victims  or  to 
governments,  or  to  some  combination  of  both.  Governments 
would  also  bear  costs  for  greatly  enhanced  security  and 
consequence  management,  including  mass  trauma  and  bum 
capabilities. 

Terrorism  obstacle  hard  to  address 

We  judge  that  terrorist  groups  now  have 
the  intent  to  attack  facilities  in  the  US  such  as  the  urban 
LNG  offloading  facility  proposed.  We  judge  that  they  could 
relatively  easily  both  obtain  the  needed  capability  and 
conduct  an  attack  on  the  urban  LNG  facility  and/or  the  LNG 
tanker  during  its  transit  of  29  miles  of  inland  waterway. 

We  judge  that  such  attacks  ran  a  high  risk  of  generating 
catastrophic  damage,  with  which  the  region  could  not 
adequately  cope  during  the  consequence  management  or 
recovery  phases. 


PROCESS  #1 


Figure  A. 27:  Analysis  Process  1  [A(1)]  Research  Note-Sheet,  Page  11  of  17 


(Appendix  A) 


88 


************  UNCLASSIFIED  ************** 


Measures  to  address  terrorism  obstacle 

We  doubt  that  deterrence 

or  prevention  measures  could  be  designed  and  implemented  for 
the  proposed  facility  and  ship  routing  that  would  be 
adequate  against  a  determined  and  skilled  terrorist  group  of 
the  type  that  exists  today.  Possibilities  for  further 
investigation,  however,  include: 

— annor  plating  the  gas  storage  containers  aboard  the  LNG 
tankers  which  transit  inland  waterways  near  populated  areas, 
—transporting  gas  along  inland  waterways  near  populated 
areas  only  in  tankers  that  do  not  freeze  and  condense  the 
gas,  thereby  significantly  reducing  the  force  and  radius  of 
an  explosion 

—constructing  structures  around  LNG  facilities  in  urban 
areas  similar  to  those  buildings  required  by  the  Nuclear 
Regulatory  Commission  around  commercial  nuclear  reactors  (an 
NRC  design  criterion  is  that  a  direct  hit  by  a  general 
aviation  aircraft  would  not  breach  the  reactor). 

All  of  those  measures  would  significantly  increase  the  cost 
of  building  or  operating  the  LNG  facility  or  LNG  tanker. 

Alternatively,  the  LNG  offloading  facility  could  be  sighted 
in  a  location  that  did  not  involve  either  an  urban 
environment  for  the  facility  or  an  inland  waterway  transit 
for  the  LNG  tanker.  Locating  the  facility  in  a  non-urban 
environment  and  eliminating  the  inland  waterway  transit 
would  significantly  reduce  both  the  attractiveness  to 
terrorists  of  an  attack  (because  the  attack  would  not 
generate  large  scale  casualties)  and  the  consequence 
management  and  recovery  burdens  on  governments  should  an 
attack  occur.  We  note  that  GAO,  the  investigatory  arm  of 
the  Congress,  recommended  in  1979  that  the  Congress  or 
Administration  prohibit  any  additional  large  scale  LNG 
facilities  in  or  LNG  tanker  transit  through  urban  areas. 

The  Federal  Energy  Regulatory  Commission  (FERC)  produced  a 
draft  Environmental  Impact  Statement  (DEIS)  of  the  Key  Span 
proposal  in  November  2004  as  part  of  the  standard  licensing 
process  for  the  construction  or  expansion  of  energy 
facilities.  The  DEIS  examined  numerous  technical  aspects  of 
the  KeySpan  proposal  and  the  effect  of  the  construction  and 
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operation  of  the  facility  on  the  surrounding  area.  The 
section  of  the  DEIS  upon  which  this  report  will  focus  is 
FERC’s  analysis  of  the  potential  threat  and  damage 
associated  with  a  deliberate  attack  on  a  LNG  tanker  in 
transit  to  the  facility,  while  docked  and  unloading,  or  an 
attack  on  the  facility  itself.88 

FERC  determined  that  the  risks  of  a  terrorist  attack  can  be 
“managed.”  While  there  is  a  level  of  risk  associated  with 
the  transport  of  any  hazardous  cargo,  the  potential 
catastrophe  that  could  ensue  from  an  attack  on  a  LNG  tanker 
or  on  the  facility  is  drastic  enough  to  merit  a  serious 
reconsideration  of  building  a  LNG  import  terminal  near  the 
heart  of  downtown  Providence. 

Alternatives  exist  to  the 

location  KeySpan  has  proposed,  and  these  alternatives  - 
where  the  damage  caused  by  a  terrorist  attack  would  be 
significantly  reduced  -  should  be  strongly  considered.  FERC 
examined  alternatives  to  the  proposal,  including  locating 
the  facility  at  a  different  site,  expanding  the  pipeline 
system,  or  building  an  offshore  terminal,  and  determined 
that  while  certain  ideas  would  eliminate  or  reduce  the 
safety  risks  associated  with  the  current  proposal,  they 
would  not  meet  the  objectives  for  improved  natural  gas 
storage  and  delivery  laid  out  in  the  current  proposal. 

Projections  to  LNG  usage 

The  U.S.  Department  of  Energy  expects  LNG  to  account  for  15%  of  U.S.  gas 
consumption  by  2025,  compared  to  1%  today.  Consequently,  LNG  imports  into 
the  U.S.  are  expected  to  grow  by  about  8.2%  a  year  over  the  coming  decade.  U.S. 
Federal  Reserve  Bank  chairman  Alan  Greenspan  testified  repeatedly  before 
Congress  that  LNG  was  the  only  solution  on  the  horizon  for  the  projected  chronic 
natural  gas  shortage. 

As  the  need  for  fuel  continued  to  rise  with  consumer  demand, 

LNG  has  become  an  increasingly  important  part  of  the  energy 
sector.  Once  a  relatively  small  market  (with  only  1  %  of  the 
total  U.S.  gas  consumption  in  2002),  expectations  for  growth 
in  the  LNG  industry  are  high.72  Consumption  is  expected  to 
increase  significantly  over  the  coming  years,  with  estimates 
of  the  total  global  LNG  trade  increasing  by  35-50%  by  2020, 
assuming  that  appropriate  facility  and  tanker  capacity  meets 
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demand.  73,74  Although  only  accounting  for  2.7%  of  the  U.S. 
energy  consumption  and  1 3%  of  total  imported  energy  sources, 
shipped  LNG  measures  over  53  billion  cubic  feet  per  year  (in 
2003).  75 

Who  might  attack 

February  11,  2006:  Q&A:  Liquefied  Natural  Gas:  A  Potential  Terrorist  Target? 
New  York  Times  "Are  LNG  ships  and  terminals  potential  terrorist  targets?  Yes, 
because  of  LNG's  raw  explosive  power,  experts  say.  Al-Qaeda,  for  example,  has 
specifically  cited  LNG  as  a  desirable  target..." 

August  12,  2005:  Terror  Alert  Offers  Specifics,  Washington  CBS  News,  The  Early 
Show  "The  FBI's  terror  warning  to  police  in  New  York,  Chicago,  and  Los 
Angeles  is  different  from  many  of  the  other  alerts  issued  by  Washington  since  the 
9-11  attacks:  It's  detailed,  according  to  Col.  Randy  Larsen,  a  CBS  News 
consultant  who  heads  the  Institute  for  Homeland  Security."  "Larsen  says  the 
main  concern  is  liquefied  natural  gas,  known  as  LNG,  which  is  so  explosive  it  can 
engulf  an  entire  building  if  a  truck  carrying  it  crashes  into  one.  " 

May  10,  2005:  Report  warns  of 'catastrophic'  risk  at  gas  terminal  Reuters  -  USA 
BOSTON  (Reuters)  -  A  terrorist  attack  on  a  proposed  liquefied  natural  gas 
terminal  in  urban  New  England  could  cause  "catastrophic  damage,"  fonner  U.S. 
counterterrorism  official  Richard  Clarke  said  in  a  report  released  on  Monday... 
READ  THE  ENTIRE  REPORT:  http://www.riag.state.ri.us/LNG_Good% 
20Harbor2.pdf 

The  shipments,  which  arrive  about  every  10  days  during  the  winter,  resumed  on 
Oct.  29,  2001,  under  the  watch  of  patrol  gunboats,  police  divers,  sharpshooters, 
bomb  squads,  and  a  helicopter.  Officials  yesterday  did  not  indicate  if  they  will 
take  the  same  precautions  this  time,  but  on  that  trip,  officials  also  halted  flights  at 
Logan  International  Airport  and  stopped  traffic  on  the  Tobin  Bridge  as  the  860- 
foot  Matthew  lumbered  into  port.  A  typical  tanker  carries  about  33  million  gallons 
of  gas. 

Distrigas  issued  a  written  statement  yesterday  saying  the  company  is 
"communicating  and  working  in  close  collaboration  with  public  safety  officials  to 
detennine  the  best  way  to  ensure  the  safety  and  security  of  our  operation  so  we 
can  deliver  critical  energy  supplies  to  our  customers  in  New  England." 

Romney  said  he  and  Menino  agreed  to  delay  the  shipment  of  liquefied  natural  gas. 
"We  have  flexibility  in  the  schedule  on  its  arrival  in  the  harbor,"  Romney  said. 

"We  plan  to  take  advantage  of  that  flexibility." 

Diane  Hernandez,  35,  a  Merrimack,  N.H.,  resident  who  grew  up  in  Chelsea,  said 
she  watched  the  tanker  arrive  last  week  when  she  and  her  mother  were  walking  on 
the  Boston  waterfront. 
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"There  were  cops  everywhere.  They  blocked  off  the  bridge,  and  there  were  at  least 
eight  Coast  Guard  boats  out  there,”  Hernandez  said.  "They  are  definitely  taking  it 
very  seriously.  They  can't  stop  everything,  but  they're  doing  the  best  they  can." 
Romney  and  Menino  said  certain  public  safety  workers  would  be  working 
overtime  until  the  alert  is  over,  but  that  there  should  not  be  many  missed 
vacations.  Romney  said  the  antiterrorist  preparations  would  cost  millions  of 
dollars,  but  the  final  tally  will  depend  on  how  long  the  alert  lasts.  Between  March 
17  and  April  17,  when  the  war  in  Iraq  prompted  an  orange  alert,  the  state  spent 
about  $200,000  a  week,  mostly  on  overtime  for  State  Police.  But  the  state 
deployed  the  National  Guard  during  that  alert,  and  Romney  said  he  doesn't 
anticipate  taking  that  step  this  time. 

#19  History  of  attacks/fires  -  Terrorism 
Although  LNG  tankers  have  not  yet  been  the  target  of 
terrorist  attack,  there  have  been  several  notable  events 
that  illustrate  the  vulnerability  of  LNG  tankers  to 
accidents  or  attacks.  A1  Qaeda  and  other  terrorist  groups 
have  shown  intent  to  target  the  energy  sector  through 
previous  actions  and  statements.36  The  rise  in  high  seas 
piracy,  discussed  in  further  detail  in  section  B  I, 
particularly  in  Southeast  Asia,  is  a  disturbing  trend. 

Additionally,  al  Qaeda  was  reported  to  have  smuggled  an 
operative  into  Boston  on  an  LNG  tanker  from  Algeria  before 
September  11,  2001.37 

History  of  attacks/fires  -  Safety  incidents 

There  has  never  been  an  accident  involving  a  LNG  tanker  that 
has  caused  significant  damage  to  the  public  or  the 
environment, 3  9  and  no  fatalities  have  resulted  from  safety 
accidents  involving  LNG  tankers.40  Through  2002,  there  were 
30  minor  safety  accidents,  which  include  collisions, 
groundings,  and  leaks.41  Of  those  30  accidents,  12  were  leaks 
that  caused  some  freezing  damages,  and  two  were  leaks  that 
resulted  in  small  vent  fires.42 

Although  LNG  tanker  accidents  have  not  resulted  in  any 
fatalities,  there  have  been  fatalities  from  accidents  at  LNG 
facilities.  Two  fatal  LNG  facility  accidents  have  occurred 
in  the  United  States  and  two  in  Algeria. 

The  following  section  catalogs  the  most  noteworthy  events.  For  a  complete  list  of 
LNG  safety  incidents,  see  University  of  Houston  Law  Center,  Institute  for  Energy, 
Law,  and  Enterprise,  LNG  Safety  and  Security,  October  2003,  p.  77-79,  available 
at 
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http://www.energy.uh.edu/LNG/documents/IELE_LNG_Safety_and_Security.pdf 

Note  descriptions  of  past  energy  infrastructure  and  attacks  on  pipelines,  etc.,  are 
included  in  Clarke  report 

As  previously  noted,  there  have  only  been  five  significant 
LNG  safety  incidents  (i.e.,  resulting  in  death)  in  either  in 
port  or  at  sea,  with  thirty  incidents  total  in  the  45  year 
history  of  the  industry.80,81  The  low  incident  rate  is 
credited  to  regulation,  improved  tanker  design  and 
technology,  improved  tanker  crew  and  ship  management 
competency.82  In  contrast,  the  Federal  Aviation 
Administration  posted  for  the  Year  2003  a  large  aircraft 
carrier  accident  rate  of  54  accidents  per  100,000  flight 
hours,  an  incident  rate  of  .3%  for  total  flight  time,  vastly 
higher  than  the  accident  rate  for  LNG  tankers. 

Types  of  LNG  incidents 

Creating  a  destructive  LNG  incident  through  an  intentional 
attack  on  an  LNG  facility  an  intentional  could  be  achieved 
through  several  means: 

•  Vaporized  LNG 

o  Using  the  gas  in  vapor  form  to  cause  physical  harm 
to  the  surrounding  population.  As  LNG  is  a 
colorless,  odorless  gas,  it  would  be  possible  to 
have  the  gas  spread. 

o  Using  the  gas  in  vapor  form  to  ignite  a  fire, 
o  Using  the  gas  in  vapor  form  to  cause  an  explosion 

•  Liquefied  Natural  Gas 

•  Using  the  liquid  gas  to  physical  harm  to  surrounding  population. 

•  Using  the  liquid  gas  to  ignite  a  fire. 

•  Using  the  liquid  gas  to  cause  an  explosion. 

The  body  of  literature  on  LNG  breaches  is  additionally 
complicated  by  the  vast  amount  of  information  that  exists 
and  by  the  fact  that  much  of  the  research  has  been  funded  or 
initiated  by  interested  parties,  either  private  companies 
with  LNG  interests  or  groups  that  stand  to  gain  or  lose  from 
the  placement  of  LNG  facilities.  In  creating  this  report, 
we  have  done  an  exhaustive  literature  search  but  in  drawing 
our  conclusions,  we  have  relied  on  those  studies  done  by 
independent  and  scientific  research  laboratories,  to  insure 
we  have  used  unbiased  findings. 
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To  date,  the  definitive  study  on  intentional  and 
unintentional  LNG  breaches  is  the  Sandia  Laboratories  report 
released  in  December,  2004.  The  report  was  designed  to  be 
the  definitive  study  that  drew  from  the  best  existing 
research.  It  examines  the  report  presents  its  own  research 
and  compares  it  with  3  additional  spill  modeling  studies 
Sandia  deems  to  be  of  sufficient  scientific  merit:  The  Lehr 
Study  (2003),  the  Fay  Study  (2003),  the  Quest  Study  (2003), 
and  the  Vallejo  Study  (2003). 

Other  obstacles 

City  Decries  LNG  Plan  says  fire  is  the  main  concern  of  residents  who  oppose 
LNG  facility,  but  also  property  values  and  local  economy  from  reduction  of  cruise 
ships,  taking  land  by  eminent  domain  (not  true),  terrorist  attack  issues  mainly 
raised  by  MIT  professor  report  with  2  mile  radius  concern,  loss  of  local 
neighborhoods  close  to  facility  through  government  buyout  of  120%  value  for 
home,  pipeline  underneath  golf  course  at  country  club 
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QUERY  SUMMARY 

Queries  to  familiarize  with  domain 

*  Liquified  natural  gas  (LNG) 

*  LNG  United  States  energy  crisis 

*  LNG  U.S.  government  agencies 

*  LNG  risks,  hazards 

*  LNG  current  policy 

Queries  targeted  at  quantifying  natural  gas  issues 

*  United  States  natural  gas  consumption 

*  U.S.  natural  gas  prices 

*  U.S.  LNG  imports 

Queries  focused  on  main  hypothesis  (local  environmental  /  safety  concerns) 

*  LNG  environment 

*  LNG  local  pressure 
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DOCUMENTS  READ  (IN  ORDER  OF  READING) 

1 .  University  of  Houston  Institute  for  Energy,  Law,  and  Enterprise.  Introduction 
to  LNG.  January  2003.  https://www.piersystem.com/clients/ 
crisis_569UniversityofHouston,IntituteofEnergy- 
INTRODUCTIONTOLNG.pdf 

2.  Energy  Resources  &  Generation  Consultancy.  International  LNG  Website. 
http://www.internationallng.com/ 

3.  Center  for  Liquified  Natural  Gas.  Liquified  Natural  Gas  Proposal.  January 
2005.  http://www.lngfacts.org/newsroom/LNG_proposal.pdf 

4.  Natural  Gas  Supply  Association.  Natural  Gas  Facts  &  Studies,  http:// 
www.ngsa.org/facts_studies/gasfacts.asp 

5.  Energy  Information  Administration/Office  of  Oil  and  Gas.  EIA  -  Natural  Gas 
Data,  Reports,  Analysis,  Surveys,  http://www.eia.doe.gov/oil_gas/natural_gas/ 
info  glance/natural  gas.html 

6.  Federal  Energy  Regulatory  Commission/Office  of  Energy  Projects.  Existing 
and  Proposed  North  American  LNG  Terminals.  January  2006.  http:// 
www.ferc.gov/industries/lng/indus-act/terminals/exist-prop-lng.pdf 

7.  Natural  Resources  Defense  Council.  Managing  America's  Latest  Natural  Gas 
Crisis.  December  2005.  http://www.nrdc.org/air/energy/fnatgas.asp 
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WHERE  DOCUMENTS  STORED  IN  FOLDERS 

LNG  General  Information 

University  of  Houston  Institute  for  Energy,  Law,  and  Enterprise.  Introduction  to 
LNG.  January  2003.  https://www.piersystem.com/clients/ 
crisis_569UniversityofHouston, Inti  tuteofEnergy-INTRODUCTIONTOLNG.pdf 

Natural  Resources  Defense  Council.  Managing  America's  Latest  Natural  Gas 
Crisis.  December  2005.  http://www.nrdc.org/air/energy/fnatgas.asp 

LNG  Organizations  Websites 

Energy  Resources  &  Generation  Consultancy.  International  LNG  Website,  http:// 
www.  intemationallng.  com/ 

Natural  Gas  Supply  Association.  Natural  Gas  Facts  &  Studies,  http:// 
www.ngsa.org/facts_studies/gasfacts.asp 

LNG  Statistics 

Energy  Information  Administration/Office  of  Oil  and  Gas.  EIA  -  Natural  Gas 
Data,  Reports,  Analysis,  Surveys,  http://www.eia.doe.gov/oil_gas/natural_gas/ 
info  glance/natural  gas.html 

Federal  Energy  Regulatory  Commission/Office  of  Energy  Projects.  Existing  and 
Proposed  North  American  LNG  Terminals.  January  2006.  http://www.ferc.gov/ 
industries/lng/indus-act/terminals/exist-prop-lng.pdf 

LNG  Issues  &  Obstacles 

Center  for  Liquified  Natural  Gas.  Liquified  Natural  Gas  Proposal.  January  2005. 
http://www.lngfacts.org/newsroom/LNG_proposal.pdf 


PROCESS  #2 


Figure  A. 36:  Analysis  Process  2  [A(2)]  Where  Documents  Stored  in  Folders,  Page  1  of  1 
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************  UNCLASSIFIED  ************** 


KEY  DOCUMENTS  (RELIED  ON  HEAVILY) 

LNG  Background  Information: 

University  of  Houston  Institute  for  Energy,  Law,  and  Enterprise.  Introduction  to 
LNG.  January  2003.  https://www.piersystem.com/clients/ 
crisis_569UniversityofHouston, Inti  tuteofEnergy-INTRODUCTIONTOLNG.pdf 

LNG  Quantifiable  Statistics  (Imports,  etc.): 

Energy  Information  Administration/Office  of  Oil  and  Gas.  EIA  -  Natural  Gas 
Data,  Reports,  Analysis,  Surveys,  http://www.eia.doe.gov/oil_gas/natural_gas/ 
info  glance/natural  gas.html 

Perspective  on  LNG  Obstacles: 

Center  for  Liquified  Natural  Gas.  Liquified  Natural  Gas  Proposal.  January  2005. 
http://www.lngfacts.org/newsroom/LNG_proposal.pdf 


PROCESS  #2 


Figure  A. 37:  Analysis  Process  2  [A(2)]  Key  Documents,  Page  1  of  1 
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************  unclassified  ************** 


HYPOTHESES  -  CONSIDERED  OBSTACLES  TO  LNG 

H 1 :  Safety  /  environmental  concerns 

H1A:  Global  environmental  concerns  (dropped) 
H1B:  Local  safety  opposition 

H2:  Cost  prohibitiveness  (dropped) 

H3:  Political  issues 


PROCESS  #2 
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************  unclassified  ************** 


COLLABORATIONS  WITH  OTHERS  DURING  PROCESS 
1 .  Analyst  Colleague:  Briefly  discussed  initial  search  strategies  /  queries 


PROCESS  #2 


Figure  A.39:  Analysis  Process  2  [A(2)]  Collaborations,  Page  1  of  1 
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************  UNCLASSIFIED  ************** 


RESEARCH  NOTE-SHEET 
A  compiled  note-sheet  was  not  used  by  this  analyst. 


PROCESS  #2 


Figure  A.40:  Analysis  Process  2  [A(2)]  Research  Note-Sheet,  Page  1  of  1 
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APPENDIX  B 

PARTICIPANT  VERBAL  REPORT  DATA 
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Study  Pilot  Participant  Interview  Notes 

Did  two  “fast  reads”  of  documents 

What  are  policy-makers  looking  for? 

-  Formatting  -  wants  a  picture  and  a  paragraph 

-  Looking  for:  what  is  the  critical  idea  that  resonates? 

If  it  doesn’t  resonate,  it's  not  pursued 

Want  30  second  sound  bites  -  for  high  speed  processing 

Similar  to  newspaper  headline  writing  style  (summary  in  first  paragraph) 

-  Method  contrasts  with  “Classic”  style  of  reporting  methodology 

“policy-maker  doesn’t  care  about  the  methods” 

Tailoring  to  audience  is  key 

-  Writing  in  best  form  for  client  to  process 

-  Uses  key  terms 

9/11  Report 

-  Not  enough  confidence  level  assessments 

Low  confidence  in  single  data  point  assessments 
Just  because  it  fits  in  with  biases  ? 

Sourcing  -  Where  did  the  information  come  from? 

-  Number  of  sources  can  give  confidence 

-  Taken  to  extreme,  every  statement  has  to  be  validated 

Problem  ~>  Solution  -->  What  to  do  about  it 

-  Decision-maker  needs  to  know  what  to  do  about  it 

-  Reason  needed  for  why  to  look  at  it 

-  One  of  two  possible  answers 

There  are  other  channels  for  rigorous  reports  that  don’t  specifically  go  to 
policy-makers 

Differences  in  policy-makers  vs.  regular  staff  (regular  analysts) 

-  put  up  articles  for  others 

-  1st  level  assessment  of  raw  data 

-  goes  into  detail  for  later  reports 

-  Senior  Staff  =  policy-makers  “right  hand”  people 

Want  things  that  are  ready  for  policy-makers 
Assumed? 

-  Issues  Managers  =  responsible  for  handling  a  single  topic  within  organization 

“Big  picture”  people  -  compile  info,  and  distribute  to  customers 
E.g.  how  could  DoD  help  border  control? 

•  Might  represent  a  commander,  but  not  in  a  supervisory  position 

•  Looks  across  stovepipes 

Should  bring  people  together,  not  make  new  tasks 


Figure  B.  1:  Study  Pilot  Participant  (P0)  Notes,  Page  1  of  5 
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-  PIA  (pee-ahh)  =  primary  intelligence  analyst  (peer) 

Make  decisions  on  what  to  do 

Stay  /  work  in  one  area  of  “junior  staff” 

-  SIA  =  senior  intel  analyst 

Tries  to  breakdown  stovepipes 

People  often  don’t  question  sources  if  things  CHANGED  -  how  do  you  know  if  they  did? 

-  Trust  can  stem  from  knowing  a  person  /  office 

-  There  is  some  inferred  trust  in  the  process 

Working  the  Area  Hard 

-  Citing  other  reports,  statistics,  current  policy  actions,  how  put  together  (flow) 

Does  it  give  what  is  needed? 

General  comments  on  reports: 

-  Briefs  “not  in  the  right  order” 

Executive  summary  +  “T-off”  Statements  (a  “hook”  to  open  a  report) 
Analysis  of  Reports: 

Report  1 

-  Too  much  background 

Never  facts,  not  specific  steps 
Stops  before  getting  to  action 

•  Too  “top  level”,  not  for  a  policy-maker 

•  Define  problem  >  why  they  care  >  what  can  they  do  about  it 

Not  “supposed”  to  advocate  policy,  but  can  be  predictive 

-  Needs  to  be  a  “newspaper”  format 

-  Never  left  the  first  level 

Some  things  are  not  really  options 
Problems  /  solutions  not  well  thought  out 

-  Too  many  obvious  questions  left  unanswered 

Implies  low  rigor  assessment 

-  Would  want  to  meet  with  author  and  specify  problems  -  “this  is  what  I  need” 

-  Not  formatted  for  policy-maker 

Needs  to  know  customer,  primary  client  -  customer  tailoring 
Must  write  well  to  get  them  to  listen 

Must  be  careful  (sympathetic)  when  dealing  w/  counter-advocacy 

Report  2 

-  Got  to  the  point  of  “why”  much  quicker 

-  Liked  the  chart  -  useful  for  policy-making 

-  Seemed  like  it  explored  the  issues  more 

-  Confident  in  getting  a  more  extensive  bibliography 

-  Seemed  like  multiple  sources  (the  more  the  better) 

-  Liked  list  of  supply  countries 


Figure  B.2:  Study  Pilot  Participant  (PO)  Notes,  Page  2  of  5 


(Appendix  B) 


104 


-  Report  had  more  “warm  fuzzies”  for  policy-makers 

-  Good  instances  of  statistics  &  research 

Clear  that  main  problem  is  lack  of  unified  policy 

Composite  Assessment  of  Report  1  &  Report  2 

-  Thought  Report  2  was  the  better  report  -  tailored  for  policy-makers 

Implies  that  the  report  is  more  rigorous 

It  looked  at  context  issues 

Would  follow-up  as  proponent  of  peer  review 

•  Would  ask  who  else  has  seen  it? 

•  Would  see  is  people  with  no  vested  interest  agree 

Intel.  Analysis  Process 

-  Number  of  peer  reviews,  and  their  responses  are  important 

Critical  to  get  feedback  however  you  can 

But  feedback  can  come  via  any  method  (still  important,  regardless) 

-  Inputs  in  intel.  aren’t  as  clear  as  going  into  a  library 

Expert  opinions  can  be  a  source  (or  e-mails) 

•  Can  add  credibility  with  an  expert  source 

•  If  no  collaboration,  trying  to  work  around  ? 

-  Sometimes  you  may  have  to  use  a  single  source 

Policy-Makers 

-  Sometimes  want  to  see  alternative  options  (“debate  can  be  healthy”) 

e.g.  “we  think  this...  other  organization  thinks  that...” 
acknowledgment  of  other  perspectives  points  to  rigor 

-  Let  policy-maker  make  the  decision 

Can’t  push  for  policy,  but  provide  information 

Comparison... 

-  Report  2  is  closer  to  being  ready  to  go,  but  would  need... 

Action  items  up  front,  summary  items 
Issues  “bulletized” 

Feedback  on  Available  Options 

-  Liked  hypotheses  considered 

-  Key  Documents 

-  Might  discuss  query  summary,  but  not  specifically  asked  for 

-  Would  only  push  beyond  #5  if  there  is  reason  to  be  suspicious 

-  Don’t  worry  about  cutting  and  pasting  as  long  as  you  cite  source 


After  reviewing  supporting  documents 

-  Report  2  didn’t  mention  counter-terrorism  paper 
Report  1  got  hung  up  on  terrorism  issue 


Figure  B.3:  Study  Pilot  Participant  (PO)  Notes,  Page  3  of  5 
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[  Felt  that  order  of  documents  did  influence  discussion  ] 

Collaboration  can  lead  you  off  the  mark  as  well 

-  Report  1  had  so  much  in  the  notes  that  didn’t  make  it  into  final  paper 

Should  “sprinkle  in  numbers”  to  show  expertise 

-  Report  2  conclusion:  need  to  solve  problem  of  economy 

-  Report  1  conclusion:  maybe  we  shouldn’t  do  it  [LNG]  at  all 

Almost  like  different  issues 

-  Found  formal  notes  to  be  useful  -  Looked  at  the  notes 

-  Report  2,  noticed  no  hypotheses  considered 

-  Found  low  rigor  sources  to  be  credible 

“Too  much  information  got  in  the  way” 

-  Report  1  was  afraid  of  picking  and  choosing  information 

Intel  analysts  serve  as  filters 

It  is  very  information  heavy,  and  this  hinders  the  flow 

-  Report  2  gets  “warm  fuzzy”  for  policy-maker 

-  Report  1  has  no  bottom  line  -  i.e.  influencing  policy 

Lost  the  client  -  e.g.  doesn’t  understand  how  regulations  is  not  relevant 
Tell  me  more  about  this? 

-  Didn’t  believe  Report  2  could  be  written  by  the  hypothetical  process 

[Technically,  it  wasn’t] 

-  Are  there  other  alternatives  to  LNG? 

Neither  of  the  papers  hit  on  the  alternatives 

-  Report  1  should  add  alternative  energy  sources  and  process  of  ruling  out 

What  are  the  counter-positions?  Identify  them. 

Articles  “wet  your  whistle”  for  alternatives  (comparing  LNG  to  other  things) 

-  Terrorism  issue  -  once  attack  occurs  it  is  not  too  late 

-  Flypothesis  list  order  indicates  importance  vs.  hypotheses  as  an  outline  in  order 

-  Saw  focus  on  terrorism  issue  as  an  author  bias  in  Report  1 

-  General  preferences 

Likes  to  see  outlines  of  papers 

Note-sheet  was  helpful  in  understanding  in  how  they  went  amiss 

•  Gave  insight  into  recovery  process 

There  is  always  the  ability  to  verbally  pull  information  from  the  author 

•  List  order  still  helps 

Outlining,  hypotheses  /  Collaboration  /  Key  Documents 

•  If  there  is  a  problem,  then  we  go  deeper 
What  are  client  requirements 


After  seeing  supporting  documents... 

-  Thought  rigor  of  Report  1  was  higher  after  seeing  process,  but  it  was  still  not  consid¬ 
ered  any  better  because  of  the  good  process.  Author  not  given  increased  credibility. 


Figure  B.4:  Study  Pilot  Participant  (PO)  Notes,  Page  4  of  5 
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-  Report  2  after  seeing  process,  it  added  more  questions  (although  enough 
sources) 

-  Report  1  process  is  flawed  b/c  data  is  there,  but  not  in  the  proper  format 

Problem  could  be  with  individual  agent  or  with  director 

-  Having  Report  1  supporting  docs,  pointed  to  a  problem  in  analyst  process... 

Did  not  increase  credibility  of  report 

-  In  general,  articles  are  reviewed  in  layers 

Usually  enough  ?  on  the  problem 

Often  not  as  concerned  with  rigor  unless  there  is  a  disagreement 

•  Rigor  is  often  flushed  out  in  the  review  process 

Actionable  Intelligence 

-  Never  enough  time  to  work  a  problem 

Often  policy-makers  “want  it  yesterday”,  even  big  complex  questions 
This  requires  a  dynamic  process 

•  What  happens  if  process  time  is  shortened? 

•  What  happens  if  2  weeks  ->  2  hours? 

•  Often  they  want  information  now  or  never... 

But  can  get  best  expert  on  the  spot 
Is  there  a  way  to  short-circuit  the  process  while  still  maintaining  rigor? 
Coordination  takes  time  -  but  can’t  let  it  suck  up  a  whole  process 

-  Often  time  policy-makers  don’t  care  as  much  if  the  information  is  “right” 

It  gives  them  somebody  to  blame  when  things  go  wrong 

-  However,  being  rigorous  is  a  good  goal 

Most  effective  on  an  “initiative  piece”  =  investigation  initiated  by  analyst 

•  E.g.  forecasting  an  emerging  problem  over  the  next  few  years 

•  Big  differences  in  time  pressures 

-  QRT  =  Quick  Reaction  Time  Process  =  squeeze  process  and  do  what  you  can 

PILOT  PARTICIPANT  DEMOGRAPHICS 

-  Has  technical  background  in  aeronautical  engineering 

1986  -  2003  as  analyst 
Last  2  years  as  a  supervisor 

Other  /  Closing  Participant  thoughts 

-  Clarifying  what  you  wanted 

-  Getting  the  analyst’s  “homework”  package 

-  Seeing  process  didn’t  change  rigor  perceptions  -  end  result  is  what  mattered 

-  Concerned  about  training  new  analysts  -  the  way  writing  changes 

-  BFC  pressures 

-  Experienced  agent’s  documents  can  look  like  Report  1  -  too  much  information 


Figure  B.5:  Study  Pilot  Participant  (P0)  Notes,  Page  5  of  5 
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Study  Participant  #1  Interview  Notes 

1)  Comparison  of  Reports 

-  Drawn  to  Report  2  initially  -  pictures  are  often  valuable 

Informal  writing  style  is  easier  to  read 
But  something  is  lost  with  this  writing  style 

-  But,  rigor  is  more  obvious  in  Report  1  -  preciseness  indicates  rigor 

-  Report  2  is  question  focused,  offers  solutions 

-  Likes  Report  1  in  that  it  quantifies  what  was  and  was  not  covered  in  report 

This  lets  policy-maker  understand  limits 

But,  would  try  to  condense  to  one  page  for  policy-maker 

•  With  option  to  provide  more  as  needed 

-  Report  1  is  ready  to  go  to  policy-maker,  Report  2  is  not 

Report  2  needs  to  be  more  precise 

•  Uses  words  like  “about”  and  “around”  too  often 

•  Makes  author  look  to  be  unaware  of  facts 

-  Drawn  to  Report  2  first  because  of  the  picture 

Report  2  could  be  given  in  a  brief 

-  Report  2  is  too  heavy  on  conjecture  (e.g.  “luckily  for  the  U.S.”) 

More  opinion  -  prompts  one  to  ask  “Am  I  reading  opinion  or  fact?” 

Report  1  gives  the  better  “warm  fuzzy” 

Bias  and  objectivity  served  as  indicators  of  rigor 

-  Condense  Report  1 

Also,  need  to  know  sources 

It  is  not  clear  how  obstacles  listed  were  determined 

-  Footnotes  could  be  useful  for  this  “form”  of  the  report 

Other  “forms”  of  report  would  include  support  material 
Footnotes  imply  academic  rigor 

•  Academic  rigor  parallels  analyst  rigor  (but  is  not  exactly  the  same) 

•  In  some  sense  “rigor  is  rigor”  -  in  line  with  David  Schum  model 

-  Question  is  very  specific  to  obstacles 

Report  1  intermixes  solutions  with  obstacles 
Report  2  goes  “off  task”  by  offering  solutions 

•  What  are  the  downsides  of  looking  for  full  solution  sets? 

But,  analyst  still  has  role  of  problem-solver  -  “here  is  a  problem,  solve  it” 

-  Report  2  proposes  policy  in  opening  paragraph,  so  tone  is  more  critical 

-  Analysis  is  more  than  just  making  reports  -  e.g.  It  also  involves  requirements 


Figure  B.6:  Study  Participant  #1  (Pi)  Notes,  Page  1  of  4 
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2)  Understanding  of  Process 

-  Wants  to  know  what  the  sources  were  --  builds  confidence 

Process  depends  on  “level  of  difference” 

-  Would  ask  questions  in  day-to-day  work  to  develop  understanding 

E.g.  ask  “What  did  you  do?” 

•  Looking  for  varied  levels  of  detail  in  response 

•  Responses  give  ideas  for  what  analyst  needs  to  work  on 

Analysis  is  not  a  step-by-step  process,  a  procedure 

•  Response  to  “what  did  you  do?”  shapes  expectations 

•  Difference  between  “procedure”  used  and  process  of  analysis 
Level  of  detail  reflects  type  of  report 

-  Wants  to  see  background  information 

Package  of  information  that  starts  with  a  summary 
Would  go  to  foundational  documents  -  see  original  reports 

-  There  is  a  working  assumption  that  “I  deal  with  competent  people” 

Assumed  that  someone  else  goes  through  verification  process  at  that  level 

•  At  their  specific  level  of  analysis 

This  is  true  even  in  agencies  that  produce  different  answers 

•  Note  the  difference  in  obstacles  between  papers 
When  you  get  to  a  point,  you  just  have  to  trust 

•  Feedback  loops  draw  out  facets  of  analysis 

•  “We  all  have  blind  spots  and  bad  hair  days.” 

-  Would  prefer  to  see  alternatives  after  potentials  in  formatting 

Gives  as  “full  of  a  picture”  as  possible 
Presumably,  e.g.  things  not  considered  in  paper 

-  Importance  of  Peer  Review 

E.g.  impacts  of  “cold  fusion”  process 

•  Lacked  peer  review 

•  Reflected  poorly  on  the  physics  community 

•  They  violated  the  process 
But,  must  go  beyond  just  peer  review 

•  It  is  not  sufficient  by  itself 

•  Peer  review  falls  apart  with  the  introduction  of  novelty 


Figure  B.7:  Study  Participant  #1  (Pi)  Notes,  Page  2  of  4 
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3)  Improving  Analysis  Reports 

-  Hypotheses  are  important 

Often  times  it  is  what  is  overlooked 

Importance  of  the  way  an  individual  relates  to  analysis 

-  Analysis  is  not  just  a  “task”  --  there  is  more  to  it  than  that 

There  is  no  one  “correct”  method 

Must  accept  that  others  have  different  practices  that  yield  solid  results 

•  And  that  their  approach  is  equally  valid 

•  You  do  not  always  have  to  completely  understand  their  method 
Cannot  always  teach  it  either 

•  Personal  experiences  lead  to  intuitiveness 

•  Which  comes  from  “reading  lots  of  stuff” 

-  “Documents  stored”  does  not  matter 

-  Sees  analogies  as  useful  to  simplify  the  world 

You  can  break  down  a  complex  thing  into  discrete,  simplified  chunks 

-  “CD  Now”  /  Amazon  approach  to  look  at  patterns  -  Who  else  looked  at  it? 

Finding  people  who  do  things  similar  to  what  I  do 


4)  Assessment  using  Process  Documents 

-  Understanding  of  Process  2  confounded  by  “thinness”  of  documents  available 

“Nothing  really  behind  it” 

Report  2  is  acceptable,  but  still  needs  something  behind  it 

•  Must  be  something  behind  visualizations  (from  Tufte  seminar) 

-  Liked  queries  to  familiarize  -  from  large  to  narrow 

Likes  to  see  a  wide  to  narrow  search  process 

-  Liked  seeing  hypotheses  -  hypotheses  are  critical 

Not  a  whole  lot  of  data  in  Report  2 

-  Liked  collaboration  insight 

-  But,  would  probably  not  forward  either  one 

Report  1  has  two  mindsets:  Energy  and  terrorism 
Liked  fact-based  nature  of  Report  1 

Report  1  process  documents  indicated  analyst  was  too  limited  (in  search) 

•  Impressive  number  of  documents... 

•  But  no  evidence  of  supports  from  other  obstacles 

-  Formatting  is  not  as  critical  unless  mandated 

-  Not  clear  in  what  order  queries  were  generated 

Did  queries  lead  to  hypotheses  or  did  hypotheses  lead  to  queries? 

-  Report  2  shows  good  collaboration 

Makes  analyst  feel  a  little  better  about  conclusions 

-  Detailed  notes  indicate  documentation,  not  necessarily  rigor 

-  Documents  lined  up  with  expectations  -  saw  more  rigor  in  Report  2 

Now  more  reluctant  to  pass  on  Report  1  as  a  result 
Would  tell  Analyst  1  to  expand  /  broaden 

-  Both  processes  had  deficient  hypotheses 

Need  to  show  that  analyst  at  least  looked  for  alternatives  -  show  an  effort 


Figure  B.8:  Study  Participant  #1  (Pi)  Notes,  Page  3  of  4 
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Can  respond  when  manager  says  “Have  you  thought  about  this?” 

-  Process  2  hypotheses 

More  broad-based  -  did  not  find  they  were  well  developed  -  feels  undone 

-  Process  1  had  good  hypotheses,  but  too  narrow 

“It’s  not  analysis  to  justify  biases” 

“Biased  study”  representative 

5)  Participant  Profile 

-  Approximately  18  years  experience 

-  No  previous  LNG  focus 

-  Order  of  documents  read... 

Did  comparative  reads,  sentence-by-sentence,  through  first  paragraph 
Then  read  Report  1  first  in  entirety 

6)  General  Thoughts 

-  Biggest  problems  in  Intelligence  Analysis 

Lacks  of  tools  to  grab  disparate  data,  convert  to  meaningful  visualizations 
Not  providing  single-point  solutions 

•  Things  pull  together  into  a  good  overview 

Often  training  is  on  how  to  use  a  tool  -  use  was  “self-evident” 

•  Problem  in  tools  if  you  cannot  use  them 

•  Usability  plus  modifiability 

-  Thoughts  relative  to  capturing  process  and  playing  it  back 

Assumption  that  people  start  with  hypotheses... 

•  But  some  start  with  requirements 
Analogous  to  Chuck  Yeager  autobiography 

•  Aflat  spin  is  bad  for  regular  pilots... 

•  But  for  test  pilots  it  was  a  good  sign  of  progress 

•  Analogous  to  “trying  to  get  to  spots  where  I  know  what  to  do” 

-  Willingness  to  document  occurs  at  different  points  in  process  of  different  people 

There  are  also  things  to  learn  from  people  who  do  things  differently 

-  How  to  support  the  briefing  assembly  process  -  idea  of  a  “keeper”  document 

E.g.  Make  it  easy  to  produce  graphics 

•  Report  1  has  a  poor  graphic,  but  it  draws  me  there 

-  Related  questions... 

How  could  notes  be  used  to  make  “data  marts”? 

Do  they  merge  with  other  documents? 
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Study  Participant  #2  Interview  Notes 

1)  Comparison  of  Reports 

-  Report  1 :  A  risk  is  not  same  as  an  obstacle  --  report  does  not  answer  question 

Obstacles  can  be  fixed,  whereas  risks  can  only  be  mitigated 

Analyst  answered  the  wrong  question  -  answer  is  not  focused  at  question 

-  Report  2:  Better  because  more  precise,  e.g.  they  identified  countries 

More  benefits  focused,  identified  few  downsides 
Listed  obstacles,  e.g.  a  lack  of  unified  policy 
Written  to  support  an  opinion 

•  Must  get  into  mind  of  customer 

-  Need  to  be  clear  about:  obstacles  vs.  risks  vs.  downsides  vs.  benefits 

-  Neither  report  answers  “how  to  get  it  done”  -  how  to  overcome  obstacles 

-  Report  2  looked  more  rigorous  --  e.g.  listed  countries 

But  does  not  go  to  that  next  “obstacles”  step 
Not  a  lot  of  “3D”  thought 

•  Answer  question,  but  then  you  think  about  it 

•  Once  you  answer  it  -  then  what? 

•  Move  ahead  and  say  “Now  what?” 

-  Report  1  wasted  pages  on  terrorism  stuff  -  Why  repeat  what  we  already  know? 

-  Getting  at  obstacles  --  what  supervisor  would  want  to  see: 

A  focus  on  policies 

Why  is  fire  and  radiation  more  of  a  risk  than  anything  else? 

Comparison  with  existing  risks 

-  Reports  had  different  interpretations 

Report  1  implied  “No”  to  LNG 
Report  2  implied  “Yes”  to  LNG 

Importance  of  Focusing  on  Customer 

-  Reporter  needs  to  be  unbiased  and  identify  biases 

-  Have  to  know  customer  to  know  what  is  good 

E.g.  reports  have  no  headers  --  should  be  clear  what  they  want  to  see 
Should  know  customer,  but  should  just  answer  question 

-  Tangent  reasoning  says  the  customer  does  not  know  the  answer 

Obstacles  need  to  be  very  dear,  can  go  back  and  clarify  question 

-  Tailor  to  executive  reader  -  senior,  no  time  to  do  a  lot  of  reading 

-  Different  versions  of  a  report  are  useful 

Everything  needs  to  be  in  an  executive  summary 

Study  Reports  Relative  to  Customer 

-  Report  2  is  higher  quality  -  shows  more  rigor 

E.g.  listing  countries  gives  more  information,  which  is  helpful 
However,  do  not  like  like  the  slant  in  Report  2  --  seems  like  marketing 

-  Report  1  is  not  focused  on  question  -  largely  does  not  apply 

-  Better  customer  focus  translated  into  “better”  rigor 

-  Would  analyst  forward  either  report  on  to  decision  maker?  No,  neither  one. 
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2)  Understanding  of  Process 

-  How  could  reports  be  fixed? 

Discussion  with  analyst,  e.g.  focus  on  issues... 

•  What  are  obstacles?  Who  has  dealt  with  them? 

•  Can  they  be  overcome?  Are  they  still  obstacles? 

-  Going  to  “second  level”  questions  is  valuable 

Follow  on  discussion  might  include... 

•  “Why  analyst  got  that  conclusion?”  and  “What  to  do  next?” 

-  Report  sources  need  to  be  identified 

-  Looking  beyond  just  the  words... 

E.g.  branching  out  to  future  energy  needs 
Doing  research  -  need  to  know  a  lot  about  it 
Relation  to  other  indicators 

-  Need  to  include  thinking  from  a  multitude  of  perspectives 

Who  are  the  enemies  and  proponents  of  LNG  and  why? 

General  Difficulties  of  Analysis 

-  Good  news  does  not  sell 

Must  be  extra  specific  when  reporting  positives 
Must  be  careful  when  presenting  view... 

•  “They  are  NOT  going  to  do  something” 

•  It  requires  more  evidence 

People  looking  for  things  that  are  bad  --  it  is  Human  Nature 

-  The  “We  report,  you  decide”  Model 

Trying  to  do  “red  cell”  activity  contrary  to  conventional  wisdom 
Not  sure  of  usefulness 

E.g.  Jill  is  a  good  mom,  but  fudged  her  time  card 
In  the  end,  CIA  has  the  final  word 

-  Looking  at  justifications  and  key  indicators 

Looking  at  what  is  important  to  me 

“Indicators  and  warnings”  --  look  for  bad  things,  want  to  see  good  things 

•  Justification  of  key  indicators  plus  knowledge 

•  Level  of  certainty  assessments 

3)  Improving  Analysis  Reports 

-  Would  go  through  process  of  analyst 

Build  up  a  “pyramid” 

•  People,  places,  problems,  etc. 

Pull  factors  to  get  a  broad  background 

-  Would  ask  how  they  did  it? 

-  Making  linkages  to  pyramid 

Is  it  analyst’s  own  view  or  taken  from  other  analysts? 

-  Always  keep  in  mind  the  question  --  focus  on  the  issue 

Build  an  understanding  of  topic  before  asking  question 

-  There  is  a  belief  that  process  can  be  inferred  from  reports 


Figure  B.1 1:  Study  Participant  #2  (P2)  Notes,  Page  2  of  4 


(Appendix  B) 


113 


Paragraphs  should  have  synergy 

Report  1  looks  like  parts  of  different  papers  --  like  cutting  and  pasting 

-  Form  /  presentation  is  key  --  sourcing  is  also  key 

But,  customer  has  final  say 

4)  Assessment  using  Process  Documents 

-  In  Report  1  terrorism  is  big  issue  --  bigger  than  it  should  be 

Paper  went  a  different  way  than  supporting  documents  indicated 

-  Found  process  documents  helpful  “in  a  way” 

-  Would  want  to  see  more  about  both  processes 

Especially  looking  at  incorporating  perspectives 
Would  form  categories  as  “for”  and  “against”  mentally 

-  Not  really  any  surprises  in  process  documents 

But,  hard  to  say  without  seeing  actual  (source)  documents 
Report  1  covered  a  bunch  of  topics  that  were  not  related 

-  Individuals  would  vary  on  how  much  they  need  to  see  about  process 

-  Documents  indicate  that  “rigor  is  still  not  enough” 

Report  1  is  more  rigorous,  but  not  good  enough 
Would  have  like  to  see  more 

-  Not  looking  for  indicators  of  rigor,  looking  for  perspectives... 

How  do  you  look  beyond  this  (question)? 

How  to  delve  into  issues  on  your  own? 

-  Looks  for  “360  Degrees  of  Perspective”,  e.g... 

Why  is  it  an  obstacle? 

Is  it  true  today? 

E.g.  UAE  port  issues  --  are  a  big  deal  now,  but  perhaps  not  before 

5)  Participant  Profile 

-  Approximately  22  years  experience 

-  No  familiarity  with  LNG  -  not  all  that  interested  in  LNG  as  a  general  topic 

6)  General  Thoughts 

-  Policy-Making  and  Analysis 

It  is  the  “Nature  of  the  Beast”  to  looking  for  things  that  are  bad 

•  In  intelligence  community,  no  one  wants  to  hear  about  good  things 

•  We  are  here  to  warn  people,  not  give  “warm  and  fuzzies” 

-  Also  important  to  say  what  you  do  not  know  -  and  also,  what  we  should  see 

-  Does  not  think  that  telling  policy-makers  what  they  want  to  hear  is  a  “problem” 

Nobody  wants  to  be  the  one  who  says... 

•  “He  does  not  have  WMDs” 

•  “He  will  not  fly  a  plane  into  that  building” 

Needs  more  information  to  decide  -  it  is  not  enough 

-  Number  of  documents  is  important,  if  it  is  known  they  were  used 

-  Would  want  to  know  the  person  who  did  analysis 

Prefers  to  talk  to  the  person  -  likes  sit-down  format 
Gets  people  to  get  at  what  is  wanted 
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-  Analysis  is  not  about  copying  and  pasting  --  must  have  understanding 

-  Questions  analysts  need  to  be  asked: 

Did  you  consider  other  alternatives? 

What  are  the  gaps? 

Who  is  against  it? 

-  Some  kinds  of  questions  can  be  used  for  most  any  task  -  could  use  a  form... 

But  not  just  a  checklist 
E.g.  use  common  questions  to  make  a  form 
Could  have  supervisor  create  “to  be  answered"  questions 
•  Supervisors  typically  have  questions  in  mind 

-  Want  analysts  to  “own”  their  process  -  do  they  believe  these  things? 

One  must  prepare  to  answer  questions  about  a  report  when  you  do  it 
-As  an  analyst,  participant... 

Uses  a  method  of  developing  the  summary  as  the  report  is  written 

-  Seeing  hypotheses  may  hinder  ability  to  be  “blank  slate"  in  critiquing  analyses 
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Study  Participant  #3  Interview  Notes 

1)  Comparison  of  Reports 

-  Report  1  was  more  qualitative  at  first  --  or  tried  to  be  (use  of  figure) 

But  there  was  not  as  much  detail 
More  casual 

-  In  Report  2,  did  not  like  use  of  “regasified”  --  took  away  from  credibility 

-  Both  reports... 

Lacked  sourcing 

Have  distracting  writing  styles 

Are  sensationalized,  which  made  reports  seem  very  opinionated.  E.g... 

In  Report  2,  9/11  is  “played  out” 

In  Report  1  there  is  excessive  focus  on  environmental  groups 

2)  Understanding  of  Process 

-  Assessment  of  rigor... 

-  Report  1  used  a  graph  -  tried  to  make  it  look  like  research  (illusion  of  research) 

Hard  to  say  they  did  more  than  just  an  online  search  to  pull  data 

-  Difference  in  focus 

In  Report  1  is  was  environmental  groups,  in  Report  2  it  was  terrorism 
But,  Report  2  also  addressed  other  issues 

-  Both  reflect  an  “unseasoned”  analyst 

Would  not  pass  either  one  on  -  both  need  editing 
Looking  for  credible  sources 
But,  Report  1  graph  adds  points 

-  Hard  to  tell  which  is  more  rigorous... 

Maybe  Report  2  because  of  broader  concerns 

3)  Improving  Analysis  Reports 

-  Revising  Process 

At  a  peer  level,  feedback  approach  would  change 

Big  issue  is  apparently  credibility  --  does  data  have  credibility? 

-  Directly  asking  -  “How  did  you  come  to  these  conclusions?” 

-  Follow  through  on  links  -  make  sure  to  do  “due  diligence” 

-  May  not  make  sense  to  ask  for  older  drafts  -  they  may  not  exist 

-  Who  else  did  they  work  with?  Resources  close  to  problem? 

Understanding  team  vs.  group  structure 

-  Both  reports  were  opinionated  --  how  could  it  be  lessened? 

Neither  report  is  unbiased 

-  Could  do  own  bit  of  quick  research  to  compare  to  the  report... 

Quick  comparative  research 

Do  not  need  to  go  to  the  library 

Use  this  information  to  judge  depth  of  report 
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4)  Assessment  using  Process  Documents 

-  Hypotheses  timeline  could  be  useful 

How  they  developed 

Did  they  just  validate  them?  i.e.  are  they  preconceived  hypotheses? 
Hypotheses  serve  as  a  window  into  mind  of  analyst 

•  Can  be  an  overlay  onto  other  dimensions 

-  Quality  of  Report  2  is  better  --  tells  more  about  the  process 

-  Found  search  terms  interesting  -  seemed  very  specific  in  search 

-  Number  of  documents  read  was  important 

-  First  hypotheses  for  Report  2  was  terrorism 

This  is  not  a  surprise 

Says  that  analyst  had  an  idea  going  in,  looked  for  confirmation 

-  Report  1  was  light  in  research,  but  started  broader 

Slightly  more  unbiased  approach 

Looked  at  less  sources,  but  started  on  the  “right  path” 

-  Seeing  reports  should  not  change  analyst’s  perspective 

-  Number  of  data  points  is  valuable 

-  Standout  points  in  processes 

Process  1  used  mostly  government  sources  (.gov  and  .org) 

Process  2  sources  mixed  in  evaluation  (.edu,  advertisement) 

However,  Process  2  had  some  articles  from  same  sources  as  Process  1 

-  Hypotheses  page  was  confusing 

Give  the  impression  of  linearity  -  i.e.  hypotheses  generated  in  that  order 
Meta-data  about  hypotheses  would  be  valuable 

•  E.g.  How  were  ideas  influenced  by  what  was  read? 

Seeing  change  over  time  is  generally  a  valuable  thing 

5)  Participant  Profile 

-  Approximately  2.5  years  experience 

-  Not  very  familiar  with  LNG  topics 

6)  General  Thoughts 

-  Citing  of  sources  is  an  emerging  global  issue 

Re-syndication  of  information  -  reissuing  content? 

Notion  of  plagiarism  is  becoming  passe 

Things  just  seem  to  emerge  -  “I  read  it  so  it  has  to  be  true” 

Likes  group  documentation 

-  Setting  up  sources 

Looking  at  issues  unbiased  -  Who  do  we  trust  to  be  unbiased? 
Demanding  practice  to  force  self  not  to  prematurely  close 

-  Could  be  more  of  a  global  problem  -  more  a  broader  reflection 

-  What  are  the  implications  on  training? 

-  People  are  unlikely  to  watch  things  over  time  -  assume  people  will  not  do  it 

-  Parallels  to  collage  artist  mock-up... 

Allows  for  seeing  meta-data  links 
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Study  Participant  #4  Interview  Notes 

1)  Comparison  of  Reports 

-  Question  is  “What  are  obstacles?” 

But,  obstacles  too  buried  into  the  middle 
Where  is  the  documentation? 

-  Form  issues... 

First  paragraph  “should”  have  the  answer  --  then  expand  from  there 

-  Report  1  seems  like  a  newswire  story  without  supporting  documents 

Sensationalistic 

Report  has  “silly”  parts  that  do  not  help  policy-maker 
Not  “technical  enough”  to  show  that  person  knows  their  stuff 

•  Not  enough  detail 

-  Report  2  has  a  first  paragraph  that  is  not  on  subject  (wants  an  answer) 

“What’s  the  point  of  price  chart  in  relation  to  question?” 

Not  obstacles  focused 

•  Talks  too  much  about  benefits  (rather  than  obstacles) 

•  Lacks  in  backing  for  some  statements 
Also,  too  much  price  focus  -  price  is  not  an  obstacle 
Environmental  comments  seem  like  rhetoric 

•  “What  does  Bush  commission  have  to  do  with  obstacles?” 

-  Report  1  is  “better”  -  it  attempts  to  look  at  issues 

Worried  about  the  rigor  “in  both  cases” 

E.g.  excessive  terrorist  focus  in  Report  1 

Terms  like  “around”  are  indicative  of  a  lack  of  knowledge 

2)  Understanding  of  Process 

-  Rigor... 

Report  1  seems  to  have  greater  rigor  than  Report  2... 

•  But  not  in  an  absolute  sense 
Graph  indicates  more  effort  for  Report  2 
Terrorist  stuff  in  Report  1  is  somewhat  off  topic 

-  Thinking  about  “next  step”  obstacles 

E.g.  need  a  radius  that  is  clear  of  houses 

-  Neither  report  is  ready  for  policy-maker 

-  Thought  analysts  should  start  over,  would  not  want  to  look  at  process 

Both  lack  obstacles 

-  Understanding  of  process  stems  from  assumed  training 

Use  of  connections  -  e.g.  collateral  information  references 
Stated  confidence 

-  Junior  analysts  often  do  not  get  structured  questions... 

Must  recognize  emerging  patterns  in  data 

3)  Improving  Analysis  Reports 

-  Face  to  face  meeting,  sending  notes  with  questions 

-  Do  not  usually  ask  about  process 
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There  are  assumed  expectancies 

-  Would  go  back  to  source  material  to  understand  process 

Although  there  are  sometimes  issues  with  translation  to  foreign  language 

-  Uses  a  general  collaborative  peer  review  process 

4)  Assessment  using  Process  Documents 

-  Report  1... 

A  lot  of  what  analyst  would  want  to  see  was  in  the  notes 
Research  documents  indicated  more  rigor  than  report  reflected 

•  Important  stuff  was  left  off  of  report 

-  Report  2... 

Rigor  of  process  showed  in  bad  document 
Process  2  indicated  not  enough  research 
“Makes  you  wonder  why  they  did  what  they  did” 

-  Seeing  process  changed  view  of  reports 

But  there  are  still  problems  with  reports 
Issues  with  sourcing  and  answering  the  question 
Allows  diagnosis  of  where  help  is  needed 

-  Report  2  showed  they  needed  to  do  more 

Hypotheses  stuff  is  missing  -  where  did  it  come  from? 

Collaboration  shows  analysts  did  his  homework... 

Otherwise,  might  not  have  seemed  that  critical 

-  Meets  a  certain  standard  on  being  a  “reporter” 

-  Report  Bias 

Report  2  did  not  really  answer  question  --  so  do  not  know  about  bias 
Report  1  does  not  show  a  great  bias 

•  i.e.  a  purposeful  slant  /  preconceived  notion 

-  Seeing  the  process  changed  assessment  from  “start  over”  to... 

How  do  I  restructure  what  you  have  done  -  re-writing  report 

5)  Participant  Profile 

-  More  than  25  years  experience 

-  Not  familiar  with  LNG 

-  Read  Report  1 ,  then  Report  2 

6)  General  Thoughts 

-  Do  not  think  analyst  should  speak  to  policy-maker 

In  an  ideal  world,  audience  would  not  matter 
Believes  NSA  is  like  that  -  no  audience  focus 

-  Policy-makers  biggest  problem  is  seeing  what  they  want  to  see 

Overcoming  preconceived  notions 

Asking  for  additional  information  until  supporting  info  is  found 

-  Design  Seeds 

Good  response  for  ability  to  tag  things  as  they  go 
Source  overlap  could  be  minimal 
Limits  to  exposure  to  raw  data 
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-  Undermining  (probably  false)  assumption:  Long  term  reporting 

Probably  mostly  events  based 

-  Looking  for  ways  to  add  to  reports,  rather  than  starting  over... 

Yet  informing  people  that  they  are  seeing  something  new 
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Study  Participant  #5  Interview  Notes 

1)  Comparison  of  Reports 

-  Report  1 :  Would  not  send  to  decision-maker 

Based  on  opinion 

Casual  writing  style  (confusing) 

Inappropriate  style  -  few  or  no  facts 
“Gratuitous  use  of  a  chart”  -  better  use  might  be  to  contrast 
Political  agenda 
Does  not  distinguish,  alarmist 
Implies  LNG  is  from  foreign  sources 
Has  to  “pull  out  obstacles” 

-  Report  2:  Better,  but  would  still  require  edits 

More  cohesive  explanations 

Clearly  states  obstacles 

Would  have  structured  differently 

“Dies”  at  the  end 

Could  have  reduced  paragraphs 

Liked  understanding  limitations,  but  not  integrated  optimally 

-  Report  2  is  better,  but  neither  is  ready 

2)  Understanding  of  Process 

-  Report  1  looks  like  a  cursory  gathering  of  information 

Not  sourced 

Feels  more  like  an  opinion  piece 
Not  very  much  rigor 
Political  slant 

Takes  a  few  facts  and  run  with  them 

No  pre-outline 

Could  be  from  a  newspaper 

Form  /  style  important  -  preference  for  “classic”  form 
Does  not  indicate  much  process  (rigor) 

-  Report  2  indicates  good  research  -  not  as  much  an  opinion 

Academic  style 

Took  time  to  think  about  obstacles 
Likes  specificity 

Given  some  structure  -  good  basic  overview 

-  Reports  indicate  different  obstacles 

Report  1  does  not  mention  terrorism  at  all 

3)  Improving  Analysis  Reports 

-  Fixing  Process  1 

Looking  for  facts  and  figures  together 
What  has  analyst  looked  at? 

Needs  to  do  more  research 

Do  not  require  chart  -  would  advise  against  using  chart 
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-  Fixing  Process  2 

Would  mainly  restructure  report 

•  E.g.  justifications  to  the  front 
Mostly  tweaking 

-  Would  like  to  see  sources 

Would  like  to  see  a  footnoted  review  copy 
Source  checking  --  public  versus  classified 

-  Might  be  interested  in  analyst  backgrounds 

-  Form  is  important 

-  Reader  must  have  confidence  in  what  is  read 

Structure  /  form  conveys  confidence 

Analyst  should  not  make  assumptions  about  reader 

Report  1  assumes  reader  holds  the  same  views  as  analyst 

•  “This  is  what  they  want  to  hear” 

•  Making  sure  personal  bias  does  not  influence  report 
Use  of  language  is  very  important 

Evaluation  of  Available  Documents 

-  Hypotheses,  Documents  Read,  and  Key  Documents  are  most  important 

Do  not  care  much  about  Query  Summary,  pending  experience  of  analyst 

-  Analyst  Experience 

Rare  to  have  access  to  note-sheet 

List  is  good  thing  to  keep  (checklist  of  process) 

Expected  to  produce  bias  in  questions 
Case-based  review 

-  Check  sheets  for  levels  of  documentation 

Should  accomplish  things  on  a  list 

Most  analysts  have  different  ways  to  accomplish  it 

There  is  not  a  “common”  process  used 

•  Especially  for  projects  with  “extra  requirements” 

Helps  to  protect  analysts 

-  In  hindsight  situations,  there  is  a  potential  need  to  recreate  process 

So  a  list  is  good  to  keep  -  chance  of  it  happening 

-  Informal  check  sheets  -  different  offices  have  different  systems 

-  Can  keep  things  in  folder  together 

There  are  some  efforts  in  progress  to  automate  this  process 

-  Difficult  to  implement  -  e.g.  query  summaries 

Could  get  some  resistance 

•  Resistance  to  keeping  track  of  things 
People  do  not  always  use  note-sheets  (maybe  1  in  10) 

Not  a  natural  habit  to  keep  record 

Time  tracking  issues  could  be  useful,  but  not  a  high  priority  initiative 
Documenting  in  figuring  out  where  things  go 
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4)  Assessment  using  Process  Documents 

-  Opinions  did  not  really  change 

-  Confirmed  suppositions  for  Process  1 

Analyst  did  not  consider  much 

•  E.g.  if  had  to  consider  hypotheses  early 

-  Good  documentation  on  Process  2 

Did  not  have  as  many  doubts  --  had  more  stylistic  questions 

-  Liked  collaboration  reports 

Do  not  often  see  things  like  this  -  people  would  not  take  the  time 
Rare  to  see  except  in  special  cases 

-  Process  reflects  on  person,  so  documents  are  useful 

-  Documentation  reflected  a  more  rigorous  process 

Not  sure  rigor  was  in  Process  1 

Research  important  and  collaboration  and  documentation 

-  Writing  style  problem  versus  issue  understanding  problem 

Wanted  to  see  more  from  Process  1 

-  Could  merge  documents  and  key  documents 

More  useful  to  separate  by  type  of  report,  rather  than  by  order  read 

-  Analyst  background  not  as  critical  in  course  of  things 

Would  come  to  know  experience  of  analyst 

-  Wanted  better  understanding  of  overlap  of  issues 

E.g.  consequences  similar  despite  intentionality 

-  Gave  confidence  that  process  was  used 

5)  Participant  Profile 

-  Approximately  23  years  experience 

-  Marginal  familiarity  with  LNG,  not  focused  on  energy 

Education  in  economics 

6)  General  Thoughts 

-  NSA  is  not  primarily  an  “all  source”  agency 

Policy  makers  may  not  see  things  in  isolation 
Taken  with  information  from  DIA/  CIA  perspective 

-  Problem 

Things  could  be  taken  out  of  context 

•  Not  taken  in  combination  with  other  “Ints” 

•  Could  be  misunderstood  or  misused  by  itself 
Output  often  not  formatted  as  sole  output 

•  Could  be  too  technical 

•  Often  does  not  bring  the  breadth 

-  Participant’s  perspective 

Analyst  can  only  inform,  not  act  -  but  can  get  the  blame 
Analysts  like  to  think  the  president  is  reading  all  their  stuff 
Analysts  often  involved  in  briefing  process 
But  cannot  get  into  details 
Building  confidence  -  briefing  on  process 
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Does  not  have  to  be  a  rigid  process 
Oversight  matter  versus  actual  concern 
Training  is  not  as  critical 
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Study  Participant  #6  Interview  Notes 

1)  Comparison  of  Reports 

-  Obvious  that  both  analysts  got  off  point  from  question 

Answering  questions  they  were  not  asked  to  answer 

-  Was  not  asked  for  risk  mitigation 

Too  detailed  on  that  portion 
Should  be  follow  up  questions 

-  Report  1  did  not  answer  question 

Gave  irrelevant  information 
Tried  to  set  the  stage,  went  off  topic 

-  Both  ignored  a  key  issue,  infrastructure 

-  Report  2  was  a  much  better  report 

Much  more  organized 
But,  still  not  ready  yet 

2)  Understanding  of  Process 

-  Report  1  has  a  graphic,  which  can  be  good 

Would  use  a  more  relevant  graph 
Graph  not  focused  on  an  obstacle 

-  Report  2  has  a  more  “organized”  process 

But  got  too  much  into  risk  mitigation 
Quantification  of  strengths  of  obstacles 

-  Question  of  what  exactly  what  analyst  is  trying  to  do  -  Documentation 

What  are  their  sources?  information?  research? 

-  Report  1  was  very  economic  focused 

-  Report  2  is  showing  more  for  process 

Does  not  specify  infrastructure  issues 
Shows  evidence  of  research 

-  Sources  also  include  collaboration 

Might  need  to  actually  see  original  report 

3)  Improving  Analysis  Reports 

-  Offices  tend  to  have  a  culture  of  how  they  do  things 

Report  1  “rambled  on”  first 
Culture  issues  in  format 

-  Access  to  their  past  work?  Consistency  --  is  it  improvement? 

Quality  control  mechanism  for  junior  people 
“This  is  how  we  do  it  and  this  is  what  we  do” 

Junior  work  subject  to  extra  scrutiny 

•  Questions  “did  you  think  of  this?” 

•  Feedback  with  senior  analyst 
Could  be  done  in  this  case 

-  Reflects  an  analyst  “not  there  yet” 

-  Also  use  feedback  from  other  analysts 

Generic  feedback  on  an  analyst 
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-  Can  assign  a  junior  analyst  to  work  with  a  senior  analyst 

Uses  a  journeyman  model 

Would  work  with  grammar  person,  then  content  person,  etc. 

•  Learning  to  turn  a  question  on  its  head 

4)  Assessment  using  Process  Documents 

-  Hypothesis  and  Research 

Assumes  that  there  are  obstacles  to  LNG 

•  Are  there  obstacles? 

•  Is  it  the  right  question? 

-  Documentation  implies  that  Process  2  is  more  rigorous 

Reflected  in  the  report 

-  Collaboration 

Shows  a  lack  of  grasp  of  the  problem 

-  Rigors  depends  on  starting  point  --  related  to  hypotheses 

Documents  show  they  dropped  topics  in  Process  2 
Did  not  show  they  focused  on  that 

•  Did  not  show  it  was  looked  into 

•  What  was  the  bases  for  dropping  hypotheses? 

Rationale  for  decisions  indicates  rigor 

•  Dropping  indicates  they  did  not  do  their  homework 

-  Research  notes  for  Process  2  were  valuable 

Seeing  order  of  queries  would  be  interesting 

-  Process  2  is  threat  of  terror  focused 

Do  analysts  get  tied  into  hypotheses? 

-  Assumption:  Document  order  read  is  reflected  by  query  order 

But  queries  and  read  order  did  not  really  jive 

-  Research  notes  and  collaboration  were  very  helpful 

Might  have  rated  collaboration  more  highly 

-  Analyst  1  seems  much  more  junior  in  approach 

Process  does  not  jive  with  “equal”  analysts 

-  Process  2  refined  queries  rather  than  searching  around  more  rigorously 

5)  Participant  Profile 

-  Approximately  19  years  experience 

-  LNG:  remembers  stuff  from  news  -  but  not  much  specific 

6)  General  Thoughts 

-  Policy  folk  are  assuming  reports  are  coming  from  “the”  expert 

-  Pressure  to  “please”  customer  --  give  right  answer 

There  often  is  no  “right”  answer... 

But  making  policies  requires  a  right  answer 

-  If  analysis  process  is  influenced  too  much,  you  have  potential  for  bad  decisions 

E.g.  Policy  makers  may  keep  asking  until  they  get  the  answer  they  want 

-  Really  just  getting  “best  guesses” 
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Study  Participant  #7  Interview  Notes 

1)  Comparison  of  Reports 

-  Report  2  is  better  than  Report  1 

It  pointed  out  areas  not  addressed  in  the  other  report 
Answered  question  more  throughly 

-  Report  1  is  “all  over  the  map” 

Not  as  well  as  written 

-  Report  1  is  not  ready  to  go  on 

-  Report  2  is  not  perfect,  but  might  be  good  enough 

Covers  issues  more  thoroughly 

-  Report  1  picked  out  “snippets” 

Looks  like  a  bunch  of  sources  and  cut  out  points 
Looks  like  first  pass  at  research  -  trying  to  get  themes 

-  Report  2  tried  to  tell  a  story 

Reflects  more  experienced  question  asking  analyst  -  reflects  experience 

-  Report  1  summarized  sources 

If  done  by  same  person,  Report  1  is  data  gathering 

-  Answering  a  question  is  looking  for  an  answer 

Do  not  want  to  see  all  the  facts 
Looking  for  answers,  not  just  facts 

-  Knowing  your  audience 

Question  as  pretty  straightforward 
Not  a  collection  of  facts  about  LNG 

2)  Understanding  of  Process 

-  Limits  of  the  analysis 

-  Trying  to  answer  a  question 

No  requirement  to  cite  sources 

Source  issues  --  Where  they  get  it?  How  current  is  it? 

-  Multitude  of  sources  in  analysis  business 

Assumed  primary  perspective  is  “own  business  line” 

•  Were  sources  in  own  business  line? 

•  Would  not  expect  sources  from  other  sources  (not  in  their  line) 

•  Work  back  to  sources 

We  should  use  those  “special”  sources 

•  Expert  organizational  perspective 
Assumed  questions  are  asked  to  whole  community 

•  This  type  of  question  goes  to  multiple  agencies 

•  Perspectives  go  across  community 

3)  Improving  Analysis  Reports 

-  Wants  to  know  what  researcher  considered  most  important 

And  where  they  come  from 

-  What  hypotheses  did  they  have  going  in  vs.  what  they  formulated  in  research 

Knowledge  of  guiding  impressions  would  be  useful 
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-  Alternative  competing  hypothesis  (ACH) 

Getting  into  the  analyst’s  mindset  as  they  work 
Looking  for  evidence  of  bias 

-  Did  not  attach  much  significance  to  how  it  was  done 

How  you  do  it  is  not  as  critical 

4)  Assessment  using  Process  Documents 

-  Seeing  the  documents  did  help 

Confirms  what  was  thought  about  quality  of  documents 

-  Report  1  showed  better  research  on  topic 

-  Report  2  was  not  so  good  at  research,  good  at  presentation 

Report  2  did  have  a  bias  --  terrorism  as  primary  hypothesis 
Document  was  heavily  biased  toward  terrorist  issues 

-  Report  2  analyst  would  be  sent  back  to  drawing  board  also 

-  Major  issues  in  reports 

Report  1  to  improve  presentation 

Report  2  to  show  less  biased  research 

Combining  the  two  would  give  different  (better)  report 

-  Report  1  is  more  rigorous  because  it  showed  a  balanced  approach 

-  But  Report  1  did  not  do  as  many  documents  --  did  not  really  collaborate 

Only  talked  to  one  colleague  -  a  far  cry  from  collaboration 
Lack  of  note-sheet  lead  to  a  weaker  presentation 

•  Report  looked  more  like  note-sheets 

-  In  Report  1  it  is  not  clear  why  stuff  is  dropped 

There  was  not  enough  about  why  decisions  were  made 
Report  needs  another  cut 

-  Liked  query  summarizes  more  -  query  is  useful  in  giving  guidance  to  improve 

5)  Participant  Profile 

-  Approximately  42  years  experience 

-  Not  much  LNG  predisposition 

6)  General  Thoughts 

-  Policy-makers  are  too  far  disconnected  from  analysis 

Removed  from  people  doing  actual  work 
If  question  /  clarification  necessary,  may  not  filter  back  to  original 
Person  who  did  work  should  always  be  included  in  work  that  goes  forward 
Too  busy  or  it  is  “beneath”  them  to  go  to  source 

-  Intelligence  Community  Network 

Things  are  passed  in  soft-copy 

Could  contain  links  that  go  right  to  originator 

•  E.g.  e-mail  the  author  -  links  right  back 

Complicated  because  there  is  not  even  a  common  intelligence  network 

-  If  it  is  admitted  it  is  a  problem,  solutions  must  be  easy 

Seniors  cannot  even  be  bothered  to  make  phone  call  -  helps  open  dialog 
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-  Too  often  intermediaries  try  to  flavor  a  report  based  on  desired  method 

Dependent  on  support  by  person  who  produced  original  document 
Too  often  documents  are  between  intermediaries  and  decision-makers 
They  hear  frequently  from  people  they  are  comfortable  with 

-  Barriers  to  getting  back  to  the  source 

Linking  from  question  to  source  could  be  valuable 
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Study  Participant  #8  Interview  Notes 

1)  Comparison  of  Reports 

-  Report  1  is  very  scattered 

Some  obstacles  are  more  like  risks 

-  Report  2  is  tighter  writing 

Has  some  statistical  information  about  cost  analysis 
Chart  helped  express  ideas 

-  Two  analysts  could  have  worked  together  to  produce  one  report 

One  to  write  it,  one  to  condense  ideas 

-  Was  not  clear  Report  2  answered  question 

-  Neither  one  is  ready  to  go  forward 

But,  Report  2  is  closer  to  being  ready 

2)  Understanding  of  Process 

-  Report  2  is  more  math  /  economic  oriented 

E.g.  Statistics  and  costs  benefit  analyses 

-  Report  1  shows  a  social  orientation 

E.g.  How  does  it  influence  people? 

What  about  risks? 

-  Report  1  was  more  emotional,  while  Report  2  was  more  factual 

3)  Improving  Analysis  Reports 

-  Would  ask  for  references 

-  Knowledge  of  analyst  background 

Qualifications  would  be  helpful 

-  In  asking  about  reports  would  be  expecting  differences  in  personality 

-  Might  require  more  development  /  discussion 

-  Not  sure  exactly  what  to  look  for 

Would  pursue  communication  via  e-mail  -  asking  questions 

-  If  deciding  between  report,  Report  2  would  be  choice  because  tighter  and  chart 

Form  of  document  better  tailored  to  decision  maker 

4)  Assessment  using  Process  Documents 

-  Key  documents  might  indicate  slant 

What  they  read  indicates  feel 

Hypotheses  also  influenced  by  what  they  read  plus  outside  influences 

-  Note-sheets  helpful 

-  Analysts  should  have  worked  together 

Report  1  indicated  heavy  research 

In  Report  2  the  style  was  good,  but  research  was  lacking 

-  Report  1  has  an  obstacles  focus 

More  rigorous  depth  of  research 
Presence  of  notes  on  many  documents 
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-  Lack  of  collaboration  is  an  issue 

There  are  benefits  of  collaboration 
And  it  is  useful  in  understanding  process 

-  Neither  one  is  ready  to  go  on 

Report  1  was  organized 
Report  2  just  pulled  information 

-  Proper  form  is  key  in  policy  decision 

5)  Participant  Profile 

-  Approximately  20  years  experience  (2  years  in  making  reports) 

-  LNG  not  an  area 

6)  General  Thoughts 

-  Biggest  problem  is  getting  information  to  policy  maker 

About  obstacles  encountered  in  doing  analyses 
E.g.  getting  across  what  was  not  covered  in  analysis 
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